How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I am looking to find the top pages based on traffic volume on my competitors websites, does anyone know of any good resources?
I want to know how which pages on my competitors websites are the most popular based on the traffic volume. I do not care how many links or directed to that page or any other metric. Only thing I am looking for is the traffic volume. It would also be nice to know the length of time spent on that page.
Competitive Research | | kanteenboy0 -
Why is my competitor's site ranking #1?
I'm about to work for a local business website that offers cleaning services and products. The keyword they want the most is ruled by a very odd site; My client's competitor's site has been around for 7 years. (Less than the average of it's competitors. Less than my client's) Has 1 backlink. Lower PA MR MT & DA than any other in the SERP. It's a 1 page site made with Flash. They do not have FB or Twitter accounts. So I thought maybe they were ranking so well because of their traffic. But neither my client, me or my coworkers have ever heard of this company. And yet, they are ranking #1. And the only thing I notice that might have helped is that the title of their page is the exact keyword and nothing more. Any ideas?
Competitive Research | | Eblan0 -
What's the best way to discover my business and search competitors?
I am trying to figure out who the real competitors are for the domain that I have been recently handed. Other than the client's references, which I don't think are the real competition (they are his benchmarks), how do I go about discovering the true competition? What the simplest, most effective way to go about discovering my business and search competitors? Given that this is a web portal, aren't both of the latter the same?
Competitive Research | | amit20760 -
Why does my site get sandboxed but this one doesn't?
I'm currently working on an entire website redesign and restructuring. My current website is extremely spammy. I really worked on it before I knew much about SEO. Now that I have a better foundation I have been remaking my site to get rid of duplicate content and to have a much better UX. With this said, I still depend on the current site to bring in my income while I'm working on getting my new site up. Google seems to constantly kick me out for about a month at a time and then let me back in. The few other times I thought it was because the new site I have been working on is on a different domain and was exactly the same for a while (I don't want google to find this new domain it's just for testing purposes). So I removed the new domain from google and in a couple weeks my main site was back in. Problem is yesterday google hit me again. I'm out of google for all my search terms. Before I was ranking on the first page for just about every "keyword + city" phrase I was trying to rank for. I can find plenty of reasons why google would give me problems with my current site. For example I have TONS of duplicate pages with just the keyword/city changed. Pretty spammy. I did buy about 5 links a year ago or so but they are pretty nothing sites. My question is if google's going to ban me why don't they just do it? They keep taking me out and putting me back in. No messages in webmaster tools. My domain is below: http://yourmusiclessons.com Also when I just started out I got the idea of "keyword + city" from a competitor that ranks for EVERYTHING which is also found below: http://www.musikalessons.com All of the links I bought were the same links the above site bought. I tried buying a link from classical.net which has a high page rank and links to musikalessons.com on each page of their site. As soon as google found the links, I was kicked out. I had them remove it. My question is musika has basically the same "spammyness" as me just on a much larger scale and yet they never seem to be kicked out of google and still rank high. I've thought about reporting musika, but I'm afraid Google will ban me as well because my site is just as bad. Again I'm working all day on fixing my site and doing a 180, it's just taking a long time and I can't launch quite yet. I live off the income from my current site though, so it's very disheartening when this happens. Any ideas would be greatly appreciated!
Competitive Research | | BrianJenkins0 -
How to track competitor who uses another website on the url?
Hi everyone, The thing is pretty simple: I have a competitor who uses a different format of url, very similar to this: www.example.com - but he started to use a thing like this: example.mtv.com The problem is that every analysis that I make tell me something about mtv.com instead of the site I want. The url is redirected. Sorry about my english, but I think it is very clear. I want to know how to track something like this, because even in competitor analysis I'm getting info about the big site, and not the one I need. Thank you very much.
Competitive Research | | bluehelmet0 -
Wondering why my site ranks well for one page and not another.
Customers mainly find our site from searching very specific part numbers on google. My site is launch3telecom.com. Take for example these two parts:
Competitive Research | | launch3
MT500A-81015 MR050-81045 Search only those parts in google - we rank 3rd for one of them and don't rank at all for another. Is there any way to check backlinks pointing to those pages? We even show up before big sites like ebay and such on one of them. Can anyone help me understand this?0 -
Why do some websites that have hardly any backlinks, poorly optimised get placed above sites that are doing all they can with SEO?
For example take these two websites. www.rfasecurity.co.uk is a site that Iam working on and have had some success with yet snifferdogs.org.uk seems to be poorly optimised with little backlinks and still gets place higher. is it due to the key word in the domian name? The key search word is sniffer dogs New at this so any help would be appreciated regards Dave
Competitive Research | | ThePod0