How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I need Ecommerce Site who have implemented J-Son Schema for Blog
Hi All, Just I need few sites who have implemented J-son schema on there blog. Looking for ecommerce blog. Hope you guys share with me good site. Thanks in advance.
Competitive Research | | amu1232 -
Many SEO tools donot allow me to check my site with https?
Hello All, Many seo tools I am using in that few tools do not allow me to test my site with https but it force me to check with http and then give report. So my query is do these tools give accurate report with http same like https? Thanks!
Competitive Research | | wright3350 -
How to top competitor's article
Hello, For bobweikel.com I'm going to work on getting some natural backlinks by making a better article than http://www.medicinenet.com/weight_loss/article.htm even though they're 30 times more of an authority than me. If you think this is a good idea, how do you recommend I top this thing? I'm just brainstorming
Competitive Research | | BobGW0 -
How on earth is this site ranking so highly?
I'm trying (& struggling) to rank for the keyword property in italy. Yet in the past 5 weeks I've seen this site -- betterpropertyitaly (dot) com -- rocket from about #30 to #2 for the keyword and am trying to figure out why. The obvious is that it's in the domain name, but that can't be the be-all-and-end-all, surely? It's a 2 yr 9 mth old site, so why the sudden spurt? I look in Open Site Explorer and the metrics look pretty weak: Domain Authority: 17 Page Authority: 29 Linking Root Domains: 9 Absolutely zero social shares. Good luck to him and all -- but I'm trying to figure out what he's doing right and I'm not. My site apart, he's beating some pretty big-hitters.
Competitive Research | | Jeepster0 -
Analysis tool that returns all possible phrases that a web site ranks for
Is there anything out there similar to the semrush.com's competitor research that can tell you a web sites total amount of appearances on Google that includes the phrases for which the domain ranks?
Competitive Research | | thebaldman30 -
Keyword Search - another site bypassed me and they are not ranked well...
I was trying to optimize the keywords "music for kids" . This is a no brainer because my URL is "musicforkids" My On-Page Report Card for "music for kids" gives my site a "A". The on page report card for the other site gets an "F". But the other site comes up number ONE in the organic Google search and my site comes up number three. The two sites above me receive a crappy on page Keyword report for the key words "music for kids" but they still come up above me for that search term. What's the deal?
Competitive Research | | musicforkids1 -
Determining why an established competitor's rankings have bombed - What's the best way to go about it?
I arrived at work this morning to find my weekly SEOmoz ranking report for a main competitor waiting in my inbox. 90% of the their rankings have tanked in the last day or so by an average of 3 pages - most down from page 1 or 2 where they had been sitting pretty for ages. I'm not in a state of (total) euphoria about this because a) you should be humble enough not to gloat at your enemy's demise, and b) I need to find out what they did wrong so that I don't make the same mistake, too. **What is your first suggested port of call to determine where my (vanquished) foe has gone wrong? How much can I find out? ** I do know one thing - with OSE I can see they've used dodgy blogging services but this, to my mind, would have been jumped upon by Google last year. No? Thanks guys
Competitive Research | | Martin_S0 -
Site Ranks Better than Comp...but why?
My site is ranking better than some of the competition sites even though the Moz Rank and Moz Trust are lower, anyone know why?
Competitive Research | | CalgaryRealtor390