How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
2 sites with low DA and PA and unoptimised <title>outranking everyone for competitive search term - How ?</title>
Im doing some keyword research for a new client, Ive found 2 anomalies for competitive search terms where sites with unoptimised <title>tags, poor backlinks and other stats, (compared to the other sites listed in the SERP) are out ranking everyone and claiming the #1 spot</p> <p> The 2 search terms are :</p> <ul> <li dir="ltr"> <p dir="ltr"><a href="https://www.google.co.uk/search?q=interior+designers+london&oq=interior+designers+london&aqs=chrome..69i57j69i65j69i60j69i65l2j69i60.2784j0j1&sourceid=chrome&ie=UTF-8">Interior designers London</a>- #1 new-id.co.uk - <a href="https://moz.com/researchtools/ose/links?site=new-id.co.uk&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a></p> </li> <li dir="ltr"> <p dir="ltr"><a href="https://www.google.co.uk/search?q=interior+designers+london&oq=interior+designers+london&aqs=chrome..69i57j69i65j69i60j69i65l2j69i60.2784j0j1&sourceid=chrome&ie=UTF-8#q=interior+designer+london">Interior designer London</a>- #1 nstudio.co.uk - <a href="https://moz.com/researchtools/ose/links?site=http%3A%2F%2Fnstudio.com%2F&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a></p> </li> </ul> <p> (interestingly they hardly rank at all for any modification of those search terms)</p> <p> The only thing i could think of is that they both had some really good links using that exact anchor text, but ive been checking OSE for the past month both in the backlinks and the just discovered but couldnt see anything like that.</p> <p>I conducted the SERP checking from a laptop with a broadband connection based in London, although im not very local to either of these locations im about an 8th of the cities distance away from one of those location and about half a cities distance away from the other. So i dont think its a local skew thats causing the SERP rankings.</p> <p>For both sites i can see they had old domains which now redirect to the them :</p> <ul> <li dir="ltr"> <p dir="ltr"><span data-mce-mark="1">New-id.co.uk = </span><span data-mce-mark="1"><a href="http://newidinteriors.com/">newidinteriors.com</a> - <a href="https://moz.com/researchtools/ose/links?site=http%3A%2F%2Fnewidinteriors.com&no_redirects=1&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a> - PA1, DA1</span></p> </li> <li dir="ltr"> <p dir="ltr"><span data-mce-mark="1">Nstudio.co.uk = intarya.com - <a href="https://moz.com/researchtools/ose/links?site=www.intarya.com&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a> - Da 32, PA43</span></p> </li> </ul> <p> Although the intarya domain was quite strong with good backlinks (but nothing above par comapred to the other sites the nstudio.co.uk site is now outranking, the anchor text was mostly natural with only some keyword anchor text from PA1 pages. </p> <p>The <a href="http://newidinteriors.com/">newidinteriors.com</a> didnt seem to have a backlink profile, both PA1 and DA1.</p> <p> Ive taken screenshots of the SERP pages below, but feel free to dig further</p> <p>Any idea what's going on here ?</p> <a download="HWkjmDf.png" class="imported-anchor-tag" href="http://i.imgur.com/HWkjmDf.png" target="_blank">HWkjmDf.png</a> <a download="A70jXse.png" class="imported-anchor-tag" href="http://i.imgur.com/A70jXse.png" target="_blank">A70jXse.png</a></title>
Competitive Research | | mike8780 -
Best way to find the highest ranking pages for a message board?
What is the best tool or method to find the highest ranking posts and content on a message board or forum? What is the best tool or method for finding the highest ranking keywords for a message board or forum?
Competitive Research | | Hondaspeder0 -
Can an https: impact (not provided) keywords in Google Analytics? Meaning if I make my site https: will it provide any more keywords. If not how can I get this data?
I have a site that I'd love to have more keyword data on. Can making my site an https: site give me more keyword data than leaving my site an http: site? Thanks!
Competitive Research | | mattdinbrooklyn0 -
Finding competitors best keywords?
How do I learn my competitor's best keywords? I realize I can add more keywords to my campaign but the problem is - I am not sure I am thinking of the right keywords. I don't want to compare with the keywords I am targeting - I want to know what THEY are targeting.
Competitive Research | | dealblogger1 -
Why would a site with virtually no external links and low authority rank very high?
Hey there: I'm not sure anyone's going to be game to answer this but... I have designed, done on-page optimization for and done ongoing content marketing and linkbuilding for a local fitness site. The site is about nine months old and I am attempting to achieve rank for the keyword "personal trainers in philadelphia." We've made progress; having taken a new site from +100 to 19 in about four or five months worth of content creation and linkbuilding on a very modest budget. I'm not complaining. What has me confused though is that there are one or two sites that are ranking very high for this keyword string with considerably shoddier metrics than our site and I'm wondering how a stock WP theme site with very few inbound links can rank top five for a reasonably competitive keyword? Our site is http://vtrainersinc.com. The site I've noticed that ranks top five and seems to have an inferior SEOMoz profile is bodycrafters.net. Anybody got any ideas as to why these things seem to occur? I have this feeling this site has stumbled upon a magic SEO elixir which I'd like to get my hands on!
Competitive Research | | splatdude0 -
Another how the *%#^ is this site ranking question
I saw a question posted by someone a while back asking how a certain (in their opinion crappy) site was ranking in the top then. It happened that there were some good reasons for that site ranking. Well.... I have stumbled on a site that seems to be ranking for (almost) no reason at all: relatively low DA/PA very few inbound links (none seem to be that special) thin content The only thing I can think of, is that the site has the keyword in the domain name. But looking at the search results, there are other domains with exact match keyword in URL and somewhat stronger metrics that don't rank.
Competitive Research | | inhouseseo0 -
Why is this site ranking #1?
I've done extensive research into http://www.maggiesottero.com/ and cannot come up to any definitive conclusions as to why this site is ranking #1 for the term "wedding dresses." The page is not optimized for the keyword. **Any thoughts? **
Competitive Research | | craigsmith3330