How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is it possible to find out how many indexed pages a website had at particular dates in the past?
I've found all kinds of websites and tools that can tell me how many pages a site has indexed currently, but how about historically? Does such a tool exist?
Competitive Research | | bigvibes9090 -
Is my client's site penalized and if not, why are other lesser quality sites ranking with lower metrics than my client's?
Hello, I'm trying to explain to a client, as to why certain websites that appear in the top 50 listings for a keyword are there when they don't have very good backlink profiles, and for this keyword, seems you need a high PA above 20 or DA above 25, just to page for page 2 - 3. Running a full SERPs report, I have seen that backlinks to the page that's ranking seems to be as high of a factor as PA, followed by DA and overall backlink profile. Having a high score in one of these 4 areas seems to increase your ranking. However, going into page 3 and beyond, seems that things kinda take a weird turn. After comparing the client's site to those ranking on page 2 I could see that all of them had better PA and more links to the page ranking for that keyword, so page 2 I understand why we don't rank on yet. However, looking at page 3, I see that the client's site in question has better PA, DA, links to page and overall links to domain. Which leaves me to think we have a penalty that's keeping us ranking 50 or beyond. Since our metrics across the boards for what MOZ is showing as ranking factors. The more odd issue is that looking at the line graph, shows right at the time we implemented sitewide https our ranking for this one keyword dropped, while rankings for our other keywords increased. Am I to assume it's a keyword density penalty? What penalties are there with a outcome of ranking limits?
Competitive Research | | Deacyde0 -
What is your review of Pushfire (aka Sugarrae's firm)?
Hi guys, I told my extended network that we're hiring an SEO firm to do some white hat SEO auditing, link building, and ongoing seo consultation. Rae Hoffman's firm Pushfire came highly recommended via several sources. They seem to have a good blend of high profile (they speak at all sorts of conferences and write at various prestigious publications) and reasonable pricing. I was wondering what fellow moz members think about this firm. Any reviews or thoughts are welcome!
Competitive Research | | Brand_Psychic0 -
Finding competitors best keywords?
How do I learn my competitor's best keywords? I realize I can add more keywords to my campaign but the problem is - I am not sure I am thinking of the right keywords. I don't want to compare with the keywords I am targeting - I want to know what THEY are targeting.
Competitive Research | | dealblogger1 -
What is a desirable range for a keyword's difficulty rating?
Hi all! I am new to the world of SEO and just getting started. As I am doing keyword research I am consistently finding difficulty ratings in the 40%-50% range. Is this considered a "par" rating or should I be looking for "a longer tail" rating? Thanks
Competitive Research | | jclayton180 -
Competitor's Ranking's Reports are Showing Blank Values for Rankings
When selecting competitors to see how they rank for specific terms in the ranking's report page here, it loads the page normally, but without any actual rankings. Where values should be, it lists all values as our competition not ranking whatsoever (for all competitors). I'm not sure if there is something we're doing wrong, or perhaps we just need to wait for a refresh, but if anybody could help out, that would be wonderful!
Competitive Research | | ACLens0 -
Raise your hand if you're doing SEO for manufacturers please.
When you're working for a manufacturer it's not exactly the same as working for a company that is flat-out informing people with content and links and revealing a lot of information (relatively speaking). Please tell me what you do when you're working with a client that is not willing to reveal too much because competition = fear of the unknown (like if you take a chance and post something online that your competition may use against you). I would appreciate less speculative and more informed answers from people who have actually done this kind of work - as opposed to those who think theoretically they may know the best way to go because they work for other types of businesses that are not as proprietary as manufacturing is. Thank you.
Competitive Research | | karlseidel0 -
Keyword based link problem on site
So I think I might have identified an issue with a site that I'm trying to get ranked for a specific keyword but, wanted to get some opinions before I started making some big changes on the site. On my homepage I have the keyword that I would like to be ranked for in the title lets say "Blue Widgets - Company Name', also on the home page I have some descriptions of our services including the keywords. I also have a couple of the keyword based links within in the content, navigation and footer. But these keyword based links all point to another page on the site: blue-widgets.htm. If I really want my home page to rank for the keyword "Blue Widgets' should all of these links point to the home page instead of the sub page? I know there are a great number of other factors that contribute to rankings but looking at my competition, this is something that they seem to be doing. The keyword based links within the content, navigation or footer all point to the homepage. I also have a higher Domain Authority than some of the sites that rank higher than me so I'm not sure if building more links is the answer. Of course I always want to build natural links but these sites don't seem to be doing that either. Any comments, suggestions or input would be greatly appreciated.
Competitive Research | | TRICORSystems0