How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's the best SEO way to benefit from your competitor's shutting down?
Hi ! Our main competitor is shutting down (website will be offline) at the end of the month and we are negotiating with them to buy their domain name: the idea would be to take advantage of their good rankings in the SERPs to redirect traffic to our pages (we're planning to crawl their site or get their sitemap and redirect th category / product pages to ours). The question is: for how long this strategy will be useful: days / weeks / months? (= for how long their pages will continue to appear in the SERPs from the day we enable the 301 redirections to our site?) Thanks in advance for your help! And if you have better suggestions, we're up to hear them of course.. 😉 Cheers
Competitive Research | | Kuantokusta0 -
Help! New site won't rank locally but should...
any help would be greatly appreciated... picked up a new client, a used car dealer in New Jersey. Have a VERY spammy site before, tons of keyword stuffing and lots of dupe content. also had a horrible design. (www.coopscars.com) We updated the design and the content, made it relevant and unique, fixed title tags, etc. Now he's not ranking for anything locally other than his business name. He's got a decent number of links, we've added relevant citations, his social signals are much stronger than they were... We do lots of SEO for car dealers, and we know that he should be ranking SOMEWHERE at this point - not saying he should be page 1, but he should at least be somewhere in the top 5... and yes, bing/yahoo are different, but he ranks over there... one page 1 for "used cars south river" - so why completely non-existant on Google? just as a test, he put up a free website at www.coopscars.net just last week - and it's already ranking for several local terms. I'm completely confused here - i'm not a noob, I know the tactics we've used on him work for other dealers. Thinking there's got to be something that's blocking him, especially since there aren't but maybe 15-20 car dealers to compete against locally and he still doesn't show up... thought i'd come over here and see if anyone has any ideas...
Competitive Research | | Greg_Gifford0 -
Can anyone suggest a AHREF's alternative?
Hello Everyone,
Competitive Research | | Juratovic
Now that AHref's has killed their keyword analysis I am on the search for a substitute.I just can't see paying $80 per month for their link analysis only even-though I do find their link index to be much much much more updated then any other company even SeoMoz (sorry Rand and company). Does anyone have any suggestions? SemRush is the only one I have seen to come close. Thank you! Pete0 -
Free tools to find country of origin of backlinks/urls
Hey are there any free tools out there which can allow me to insert a large list of urls, and it determines the country of origin of the domain. I know the paid version of majestic does, but i was wondering if theres any free tools? Cheers, Chris
Competitive Research | | monster990 -
What is the best tool to monitor your competitor's backlinks?
I am looking for a tool that not only accurately tells me how many links are being built by a competitor, I want to know the specific URLs for specific periods. If I can filter/search by time periods, i.e which links were added for the month of August, 2012, that would be great. Thanks for the help.
Competitive Research | | inhouseseo0 -
I don't get it
Hi...I'm new here....not a professional SEO at all....just been teaching myself as much as I can about SEO on my own as I'm a small business owner desperately trying to make something out of herself. Could use your thoughts on this...please forgive me if this isn't your usual type of question you get here: so...I've been going through all the suggestions for on page optimisation and researching competitors links given through this lovely SEO Moz pro account . One competitor in particular....has spammed A LOT of sites...I think she's hired some company to do it for her. There's article submissions that don't even sound like they are written by someone who speaks English and very low quality sites for back links including some with adult content...however she's got thousands of these links so she ranks extremely high on all keywords. All her pages have very high keyword density and could be accused of keyword stuffing big time. I put her website through the SEOmoz grader....she's got a C and I've got an A. She's completely catered her site to Google not the customer and its obvious. But other competitors in my product have started doing the same thing as this gal and low and behold their sites are popping up in google searches also. Beginning to feel frustrated with my hours of efforts and wondering how I can compete with people like this when I'm trying to be a good girl with Google and focus on creating a great site. My hits per day are increasing slowly, my Alexa rank (hoping this matters) is improving rapidly and you can actually find me on the google search when I couldn't before (I'm page 10 now yay) so I don't feel like a total failure but still am wondering if hitting page 1 for my keywords will happen this lifetime. Why does Google seem to reward people who go against all these countless books and resources on SEO I'm reading? Could use any thoughts/suggestions you might have on this matter. Thanks for your help. x
Competitive Research | | ldnwickless0 -
Sites With Duplicate Content Ranking Way Higher
The site I am writing about is easendtorontohomes.com I don't get it. I see that some of my competitors have more inbound links and have been around longer. Their SEOmoz trust etc is also higher. But I don't get how many could possibly be ranking higher since they're sites are all template based site via Web Tech Design and the content is duplicated from one to the other. The other weird thing is that they do everything wrong - they stuff their keywords tag, some of them use the same keywords/meta data on every single page, some of them don't even have meta tags, some of them don't even have a blog with ANY original content. Apart from listings EastEndTorontoHomes.com has TONS of original, well-researched, and keyword rich content. I just don't get it. Is it possible that google has some sort of relationship with that company? Or am I totally missing something? Some of these sites include: tinasmith.ca, mikeclarke.com, teamkassen.com
Competitive Research | | annasus0 -
How due I take the advanced report that i have exported as a excel file and recreate your online color coded version with the data.
In the keyword difficulty screen how do I recreate your beutiful color coded spreadsheet in the exported excel file based on the advanced reports.
Competitive Research | | bmcfarlane0