How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I find out the exact match visitors for a competitor's website?
Hi, I'm trying to find a way of finding out the estimated number of exact match visitors that a competitor's website has got. I know that SEO agencies can find this out, and there must be a way of doing this for myself. Thanks
Competitive Research | | MThacker0 -
Penguin Case Study: Need sites that were hit
I am currently trying to do a massive study on Penguin right now, you can read more about it here if you wish http://www.seomoz.org/q/my-penguin-recovery-attempt Anyways I need some data, can anyone who is willing give me a list of known sites that were hit from Penguin 1.0 Only. This can include the 2 Data refreshes after that, but basically nothing from 2.0 I also need a site in the same niche that you are aware of that did not get hit from Penguin 1.0. Possibly the #1 guy at the top would be nice. So I need Domains Hit from Penguin 1.0 Top Domains in same niche that were not hit You can send me a message with the domain if you are not comfortable posting it here. Thanks all ahead of time 🙂
Competitive Research | | cbielich0 -
What's the value of Exact Match Keyword Domains vs. Company Name Domains?
Hey Mozers, I was in a discussion this morning about the value of Exact Match Keyword domains vs. a company name domain and wanted to get a little more clarification. Let's say we are doing a site for a company called Favored Dental, and they have had the domain favoredental.com for quite a while and have their authority built up in it. Is it better to have favored-dental.com or favoreddental.co or keep its current form? The reasoning behind the alternate domains would be they have the exact match keyterm, in this case lets say "Favored Dental" is the keyterm we were going after. To my knowledge EMDs aren't as relevant as they'd use to be as Google would rather branding of companies instead of keyterm domains? Is this correct, or do EMDs of keywords you're going after hold higher authority? Thanks for the clarification!
Competitive Research | | MonsterWeb280 -
Analyzing Back Links - Says site A has back link to Site B but when I look at site B I can't find any back link to Site A. Why?
I am new to SEO Moz - It looks like incredible technology. I was playing around with different websites to see where they had back linked to see how it works. Looked at a site called racingsecretsexposed [dot] come and it said that it had dozens of links to www.ndesignstudio.com such as: ndesign-studio.com/blog/best-wordpress-sites?replytocom=893 with link anchor text "laying horses" but when I do a search for the company name, or the anchor text "laying horses", or the owner of the company's name on ndesignstudio.com - nothing appears. Why not? Isn't the back link anchored by the text laying horses, which should link back to the racing secrets website? Thanks
Competitive Research | | NewtoSEO900 -
Keyword based link problem on site
So I think I might have identified an issue with a site that I'm trying to get ranked for a specific keyword but, wanted to get some opinions before I started making some big changes on the site. On my homepage I have the keyword that I would like to be ranked for in the title lets say "Blue Widgets - Company Name', also on the home page I have some descriptions of our services including the keywords. I also have a couple of the keyword based links within in the content, navigation and footer. But these keyword based links all point to another page on the site: blue-widgets.htm. If I really want my home page to rank for the keyword "Blue Widgets' should all of these links point to the home page instead of the sub page? I know there are a great number of other factors that contribute to rankings but looking at my competition, this is something that they seem to be doing. The keyword based links within the content, navigation or footer all point to the homepage. I also have a higher Domain Authority than some of the sites that rank higher than me so I'm not sure if building more links is the answer. Of course I always want to build natural links but these sites don't seem to be doing that either. Any comments, suggestions or input would be greatly appreciated.
Competitive Research | | TRICORSystems0 -
Should I link to this site or is it too spammy?
Hi all, I've been asked if I want to exchange links with http://www.myteetimesonline.com/ and I'm tempted but fearful that its link profile looks spammy. What do you guys think? I'm basically starting from scratch with my own site http://www.golfgear4you.com and have a Page Rank 0 and Domain authority 32. The guys that want to link to me are Page Rank 3 but Domain authority 35. Their content would be of interest to my site's users and vice versa but my biggest concern is their link profile looks awful because the sites that link to them don't appear to be very relevant. I've done a lot of reading on this and have found the resources on this site to be a goldmine. The generally message is avoid easy links to spammy sites because they could hurt credibility. I just want to check I've actually learnt something and my instincts about this site are correct. i.e leave it alone!! Thanks, Andy
Competitive Research | | getzen560 -
How can I estimate a domain's overall organic search traffic - any tools?
Most of my analysis revolves around looking at rankings for specific keyword phrases that I've identified as important/relevant. But it'd be nice to be able to look at a domain and get a sense for how much organic traffic they get overall. If they're not ranking for the keywords I'm researching but have a lot of organic traffic that would be a nice signal to me that they are probably targeting other phrases more or have a big brand presence or something. Any suggestions? Thanks! Jeff Gibson
Competitive Research | | jeff.gibson0 -
Government Sites Cluttering Results?
Hi Guys, Have you ever come across government sites that are cluttering up SERP's that you're trying to rank for? For a new site that I'm working on one of the keyword terms is "driving test cancellations" and is in the UK. 3 of the top 4 results are government related sites which have verry little (if not nothing) to do with the keywords. Whilst these government sites are (very) loosley related to the keyword terms, and understandably have high pa/da, what would be the best way to try and rrank higher than these sites. I'm in the process of building links and social profiles - I'm really just wondering if there's something I'm missing that is an "easy fix" for jumping ahead of these sites - or getting them removed due to their lack of relevence. Gary...
Competitive Research | | perfectweb0