How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I can see competitors ranking for certain long-tail keywords but cannot find them on web pages. What am I missing?
Hi there. I'm pretty new to SEO and I've been doing a fair bit of training but there is one aspect I have yet to grasp. When I carry out keyword research, I get all these results and I understand the metrics. What I'm not getting is, when a competitor is ranking highly for say "where can I buy fresh turkeys", I assume that that phrase must appear somewhere on the page, but it doesn't. I realise I'm just not thinking about this in the right way. Can anyone offer clarification, please? Kind regards, Bruce
Competitive Research | | BruceBarbour0 -
Is it possible to find out how many indexed pages a website had at particular dates in the past?
I've found all kinds of websites and tools that can tell me how many pages a site has indexed currently, but how about historically? Does such a tool exist?
Competitive Research | | bigvibes9090 -
What's the value of Exact Match Keyword Domains vs. Company Name Domains?
Hey Mozers, I was in a discussion this morning about the value of Exact Match Keyword domains vs. a company name domain and wanted to get a little more clarification. Let's say we are doing a site for a company called Favored Dental, and they have had the domain favoredental.com for quite a while and have their authority built up in it. Is it better to have favored-dental.com or favoreddental.co or keep its current form? The reasoning behind the alternate domains would be they have the exact match keyterm, in this case lets say "Favored Dental" is the keyterm we were going after. To my knowledge EMDs aren't as relevant as they'd use to be as Google would rather branding of companies instead of keyterm domains? Is this correct, or do EMDs of keywords you're going after hold higher authority? Thanks for the clarification!
Competitive Research | | MonsterWeb280 -
Keyword Search - another site bypassed me and they are not ranked well...
I was trying to optimize the keywords "music for kids" . This is a no brainer because my URL is "musicforkids" My On-Page Report Card for "music for kids" gives my site a "A". The on page report card for the other site gets an "F". But the other site comes up number ONE in the organic Google search and my site comes up number three. The two sites above me receive a crappy on page Keyword report for the key words "music for kids" but they still come up above me for that search term. What's the deal?
Competitive Research | | musicforkids1 -
How do i improve my site's google ranking against my competitors?
the searches that i would like to improve my ranking on are
Competitive Research | | oswaldjr
st kitts real estate
nevis real estate
st kitts and nevis real estate. my website is www.sknlistings.com0 -
How do I find the best converting keywords
I rank decently on local search for some of my keywords but find that the traffic coming to my site is not from those keywords. I also find that they never convert to sales queries. Is someone able to help me out with that? Thanks My domain is www.hykano.com
Competitive Research | | Skwtayler0 -
How many Page Rank 8 sites are there?
Anyone have any estimate of how many Page Rank 8 sites are out there? Best, Steve
Competitive Research | | Aggie0 -
Why there is no even close correlation between MajesticSEO data and Open Site? Explorer?
I compared my site home page against my competitor homepage with Open Site Explorer and I found that according Open Site Explorer I have only 3 backlinks where in MajesticSEO I have more than 100 backlinks in 'Historic' data and more 90 in '30 days fresh data'. Why there is no even close correlation between MajesticSEO data and Open Site Explorer? How SEOMoz count backlinks?
Competitive Research | | eladlvy0