How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I can see competitors ranking for certain long-tail keywords but cannot find them on web pages. What am I missing?
Hi there. I'm pretty new to SEO and I've been doing a fair bit of training but there is one aspect I have yet to grasp. When I carry out keyword research, I get all these results and I understand the metrics. What I'm not getting is, when a competitor is ranking highly for say "where can I buy fresh turkeys", I assume that that phrase must appear somewhere on the page, but it doesn't. I realise I'm just not thinking about this in the right way. Can anyone offer clarification, please? Kind regards, Bruce
Competitive Research | | BruceBarbour0 -
Why has my authority article gone from P1 to below the top 50? Along with the rest of my site...
My site has run into some serious and unexplained SEO issues over the last 6 months, i.e: I wrote the content at http://www.flatroofs.co.uk/flat-roof-insulation/ about 3-4 years ago, and ever since it's competed for position with the huge PLC websites and hovered around the top 3 spots in google.co.uk. However, since May this year things have declined rapidly and now isn't even in the top 50 for [flat roof insulation]! Why?? I am aware that this content has been pretty heavily plagiarised since day 1, with some other sites making the effort the 'rehash' my wording and others just blatantly copying. At the same time it's also been relatively well linked to (for a very boring subject) since day 1. Over the same period (since May) the site has suffered a huge decline in traffic and rankings, as can be seen here http://www.flatroofs.co.uk/wp-content/uploads/2013/11/Capture.png, but I find the huge drop harder to justify for a relevant article than for the homepage/site in general. What strikes me about the above is the two step changes in traffic volume; one in early June and the other in late October. I am vaguely aware of the penguin update etc, but it seems to me like I've been manually banished somehow? Thanks in advance for any advice!
Competitive Research | | Biota0 -
Analysis tool that returns all possible phrases that a web site ranks for
Is there anything out there similar to the semrush.com's competitor research that can tell you a web sites total amount of appearances on Google that includes the phrases for which the domain ranks?
Competitive Research | | thebaldman30 -
Why does my site get sandboxed but this one doesn't?
I'm currently working on an entire website redesign and restructuring. My current website is extremely spammy. I really worked on it before I knew much about SEO. Now that I have a better foundation I have been remaking my site to get rid of duplicate content and to have a much better UX. With this said, I still depend on the current site to bring in my income while I'm working on getting my new site up. Google seems to constantly kick me out for about a month at a time and then let me back in. The few other times I thought it was because the new site I have been working on is on a different domain and was exactly the same for a while (I don't want google to find this new domain it's just for testing purposes). So I removed the new domain from google and in a couple weeks my main site was back in. Problem is yesterday google hit me again. I'm out of google for all my search terms. Before I was ranking on the first page for just about every "keyword + city" phrase I was trying to rank for. I can find plenty of reasons why google would give me problems with my current site. For example I have TONS of duplicate pages with just the keyword/city changed. Pretty spammy. I did buy about 5 links a year ago or so but they are pretty nothing sites. My question is if google's going to ban me why don't they just do it? They keep taking me out and putting me back in. No messages in webmaster tools. My domain is below: http://yourmusiclessons.com Also when I just started out I got the idea of "keyword + city" from a competitor that ranks for EVERYTHING which is also found below: http://www.musikalessons.com All of the links I bought were the same links the above site bought. I tried buying a link from classical.net which has a high page rank and links to musikalessons.com on each page of their site. As soon as google found the links, I was kicked out. I had them remove it. My question is musika has basically the same "spammyness" as me just on a much larger scale and yet they never seem to be kicked out of google and still rank high. I've thought about reporting musika, but I'm afraid Google will ban me as well because my site is just as bad. Again I'm working all day on fixing my site and doing a 180, it's just taking a long time and I can't launch quite yet. I live off the income from my current site though, so it's very disheartening when this happens. Any ideas would be greatly appreciated!
Competitive Research | | BrianJenkins0 -
How can I find the pages on my site with the highest page rank?
I have a lot pages on my site and I'm wondering if there is a tool out there that will find all the pages with the highest pagerank for me? Also, it would be helpful to find that on competitors sites as well. Any ideas?
Competitive Research | | shawn810 -
Our site being outranked by competitors with lower "moz" scores - due to on-page SEO?
Howdy, Our SEO efforts are doing well, but for a few keywords it seems we cannot budge one of the competitors sitting in spot #1. Through some competitive analysis I've noticed that our website has a much higher mozRank with regards to both page and domain compared to the current #1 spot. My question is what kind of factors could be the issue as to why we are still being outranked. Is it simply a case of poor on-page SEO at this point or should I be taking the mozRanks with a grain of salt.
Competitive Research | | marketingdepartment.ch0 -
Twitter as a website's #2 ranked linked page?
A site I'm researching on open-site explorer has a #2 link with page authority of 52 and Domain authority of 97, and that link is the site's twitter page. No other sites I've researched have had their twitter page show up in it's link rankings like this, can someone explain?
Competitive Research | | TheSquareFoot0 -
Government Sites Cluttering Results?
Hi Guys, Have you ever come across government sites that are cluttering up SERP's that you're trying to rank for? For a new site that I'm working on one of the keyword terms is "driving test cancellations" and is in the UK. 3 of the top 4 results are government related sites which have verry little (if not nothing) to do with the keywords. Whilst these government sites are (very) loosley related to the keyword terms, and understandably have high pa/da, what would be the best way to try and rrank higher than these sites. I'm in the process of building links and social profiles - I'm really just wondering if there's something I'm missing that is an "easy fix" for jumping ahead of these sites - or getting them removed due to their lack of relevence. Gary...
Competitive Research | | perfectweb0