How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Finding most popular sites within a category - who offers that information?
Hello Mozzers - I am researching the most popular sites within a category - are there any services that identify the most popular sites within a category - I noticed Alexa do offer a service but I have no idea who else offers such a service (Majestic, perhaps?). Thanks in advance, Luke
Competitive Research | | McTaggart0 -
What to do about one site dominating search results? (multiple pages ranking)?
Anybody have thoughts on dealing with search results where the same site gets listed multiple times? "weebly vs wix" is one example (same site #1-3, repetitive articles, not crazy high authority), but I see this now and then. I know Google likes variety, so it's weird for me to see results like this dominating search results. Thoughts? What gets these sites to take over the top rankings for a specific term? Any way to rise up in this situation, outside of the usual? Any tips on duplicating this kind of success?
Competitive Research | | davidwaring0 -
SEO's done, 301s in place, old site STILL outranks new site. What to do?
Since Sep 2010 I have had a site up with minimal SEO optimization (www.chrisbrushmusic.com). Oct 29, 2012, I launched a new site on a new domain (www.chrisbrushdrums.com) with more content and tons of SEO work behind it. The content of the new site is significantly different from the old site, and I wish to keep the old site around. I have 301's in place for specific URLs on the old site that point to the new site. I have submitted xml sitemaps for the new site. As of now, the old site still outranks the new site (i.e. Google search for "nashville session drummer" and my old site is #9 - my new site is nowhere). What should I do? Thanks.
Competitive Research | | cbrush0 -
What's the best way to discover my business and search competitors?
I am trying to figure out who the real competitors are for the domain that I have been recently handed. Other than the client's references, which I don't think are the real competition (they are his benchmarks), how do I go about discovering the true competition? What the simplest, most effective way to go about discovering my business and search competitors? Given that this is a web portal, aren't both of the latter the same?
Competitive Research | | amit20760 -
How did this little site rank #2?
Keyword: "adoption agency" Site: http://www.opendooradoption.org These guys are ranked #2 for this very competitive term in all US regions. They're beating Bethany (and my company) who are orders of magnitude larger in both company size and link profile. Their domain authority is 40, and they have less than 200 links pointing to them with "adoption agency" in the anchor according to OSE. Their on-site is pretty much nonexistent. Does anyone have any theories as to how they accomplished this?
Competitive Research | | AdoptionHelp0 -
Good tools to compare site speed on different sites?
Google tells me that my site is slower than 81% of the internet.Now this is a very large news website, so I would like to compare this against other very large news websites. Are there any tools that will be able to tell me the speed of a list of websites? I am looking for something a little more detailed than 'this site loaded in 1 second'. Also, do any of you fellow SEOs know of any good tools to analyse what is causing your site to slow down? Kind regards
Competitive Research | | MirandaP0 -
How can I obtain analytics and social data for a site that I'm not authorized on?
I'd like to get traffic information and facebook /twitter stats for a company. Don't know their ua, logins or have access to any of their pages. Where can I find this information? Any help would be greatly appreciated!
Competitive Research | | brandonschultz0 -
Search shows site links, but not hyperlinked?
2 of our competitors on Google have site links show in their results which brings you to that specific page. In our listing we show up with the same results a few spots behind them with our site links visible showing very similar content, but are not actually hyper linked. Is there a way to fix the issue and let people actually be able to click on our site links instead of just reading them?
Competitive Research | | sknott0