How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Non ecommerce site with high DA, being outranked by lower DA sites with ecommerce - help?
I have a website that we look after, up until January this year the rankings were really strong - the site has a much higher domain authority than the competitors, much more links and much more relevant content (all measured with moz's tools). About 6 months ago the rankings started to dive for national search terms. Now one thing ive noticed is that nearly all of the other sites in the search sell products online where we have no such function - could this be correlated - has anyone else experienced this?
Competitive Research | | rhysc1 -
What to do about one site dominating search results? (multiple pages ranking)?
Anybody have thoughts on dealing with search results where the same site gets listed multiple times? "weebly vs wix" is one example (same site #1-3, repetitive articles, not crazy high authority), but I see this now and then. I know Google likes variety, so it's weird for me to see results like this dominating search results. Thoughts? What gets these sites to take over the top rankings for a specific term? Any way to rise up in this situation, outside of the usual? Any tips on duplicating this kind of success?
Competitive Research | | davidwaring0 -
Is my client's site penalized and if not, why are other lesser quality sites ranking with lower metrics than my client's?
Hello, I'm trying to explain to a client, as to why certain websites that appear in the top 50 listings for a keyword are there when they don't have very good backlink profiles, and for this keyword, seems you need a high PA above 20 or DA above 25, just to page for page 2 - 3. Running a full SERPs report, I have seen that backlinks to the page that's ranking seems to be as high of a factor as PA, followed by DA and overall backlink profile. Having a high score in one of these 4 areas seems to increase your ranking. However, going into page 3 and beyond, seems that things kinda take a weird turn. After comparing the client's site to those ranking on page 2 I could see that all of them had better PA and more links to the page ranking for that keyword, so page 2 I understand why we don't rank on yet. However, looking at page 3, I see that the client's site in question has better PA, DA, links to page and overall links to domain. Which leaves me to think we have a penalty that's keeping us ranking 50 or beyond. Since our metrics across the boards for what MOZ is showing as ranking factors. The more odd issue is that looking at the line graph, shows right at the time we implemented sitewide https our ranking for this one keyword dropped, while rankings for our other keywords increased. Am I to assume it's a keyword density penalty? What penalties are there with a outcome of ranking limits?
Competitive Research | | Deacyde0 -
Why Won't Google Update My Title?
I have waited plenty of time. Google has cached several pages with the updated title placed in the <title>tags.<br /><br />However search results, continue to show otherwise.<br /><br />I read the following (https://support.google.com/webmasters/answer/35624?hl=en)</p> <p style="padding-left: 30px;"><em>If we’ve detected that a particular result has one of the above issues with its title, we may try to generate an improved title from anchors, on-page text, or other sources. However, sometimes even pages with well-formulated, concise, descriptive titles will end up with different titles in our search results to better indicate their relevance to the query. There’s a simple reason for this: the title tag as specified by a webmaster is limited to being static, fixed regardless of the query. Once we know the user’s query, we can often find alternative text from a page that better explains why that result is relevant. Using this alternative text as a title helps the user, and it also can help your site. Users are scanning for their query terms or other signs of relevance in the results, and a title that is tailored for the query can increase the chances that they will click through.</em></p> <p>The reason I want to change my title, is because there seems to be a relevancy issue (as pointed out my other community members here.) Google is having trouble recognizing understanding what our site is about.<br /><br />So instead of a title that reads, "Felix And Fingers: Dueling Pianos" (as Google continues to use) I prefer "Dueling Pianos - Felix And Fingers"<em> </em> I don't believe Google is recognizing us correctly as a dueling piano company.<em><br /></em>Google doesn't seem to like that. Any idea why or how I might go about getting this updated?</p></title>
Competitive Research | | osaka730 -
Site Architecture for a Services Company
Hey everyone, I'm doing the keyword research for a new virtual assistant services website that I am building. Being that there's almost a limitless number of services that a customer can outsource to a virtual assistant, I have a lot of service keyword options. Here's the problem: I don't know how to layout my services in a scalable fashion so that I can continue to add onto the services list (each with their own unique and helpful page) after the website is published. I have two options: 1. Create a service page with categories, each with links to the individual services. Like on my competitor: http://www.virtualemployee.com/services but with links to pages focused on the individual service keyword (ex. virtual receptionist services). 2. Don't have a services page, but include the services as blog posts like on my competitor's website: http://www.virtualstafffinder.com/blog. Obviously, the blog would be much more "scalable" in the terms of adding more services. It also reduces a lot of clutter on my website. However, it's not categorized like the average service or e-commerce website. Not sure if this is necessarily a bad thing. What do you think? Your help would be greatly appreciated!
Competitive Research | | Stryde0 -
Confused about sites ranking when compared to each other
I'm looking at the keyword "snow guards" for two competitor websites: snogem dot com (SG) snoshield dot com (SS) I've been watching these two for about 4 months. SG stays in the top 5 SERPs consistently, while SS bounces around from 15-20 on Google. I've looked at the competitive link comparison and SS appears to have a much better link profile when it comes to DA. Page specific metrics in OSE show SS with a higher mozrank, moztrust, internal followed links, linking root domains, total linking root domains, and linking C Blocks. It appears to me that SG has many more links, but their links seem to be many site wide exact match keyword links, and paid for advertisements on a major site in the metal roofing industry. Honestly the linking profile of SG surprises me that they rank so much higher than SS after the Panda/Penguin updates. I have two questions: 1. When competing against these sites, should I model SG's link profile? I'm nervous to because of my previous statement thinking they will get hit with Panda/Penguin like updates. 2. What would it take to bounce SS to competing with SG on the first page? Is the number of links the only thing this site is missing, or is there something else to focus on.
Competitive Research | | kadesmith0 -
Working through the slides on my first seo project, I'm trying to determine the most popular content on competition sites.
From the video, I understand I should be looking for unique opportunities. But is there a baseline for product qualification, ie., a min. PA,LRD, Inbound LInks, social shares, etc. I guess I'm asking what I should be looking for in the data provided on each competing site? What determines THE most popular and unique product?
Competitive Research | | AhmadS0 -
Crazy SEO question (maybe I'm missing something?)
OK - so one of our customers just called us and told us an interesting story: A local SEO company called her yesterday to try to sell their services to her. She's in the process of starting SEO services with us, so she told them she wasn't interested. The sales guy told her that they were better (without even asking who she was currently using) and asked her for a term that she'd like to rank higher for. She said she'd like to rank higher for "spray in bedliners northern ky" and he said "Gotcha, call you tomorrow" He called back just now and told her to look at Google. She's now ranking number one for that term. He didn't have access to her site, so he wasn't able to change anything on her site. He won't tell her what he did, and told her it was legitimate - but it seems to me that with only off-site tactics, it'd be nearly impossible to white-hat her site to number one overnight... Any ideas what he's doing? First of all, we want to be able to tell her what he's doing, because she's curious. More importantly, we want to be sure he's not doing anything black-hat that's going to hurt our client's site. Thanks for your help, Mozzers!
Competitive Research | | Greg_Gifford0