How to Find Another Site's robots.txt File?
-
An SEO report, not by SEOmoz, says my top two competitors have robots.txt files that disallows spidering. I suspect that their robots.txt file doesn't disallow all spidering.
How do I find out what is in their robots.txt files?
-
Should I disallow 80legs and sitebot robots?
Why might these two robots be disallowed?
-
User-agent: 008 Disallow: /
(Tells 80legs Robot to stay out of the website)
User-agent:* Disallow: ``` (Tells all other robots to visit all files of the website) 2) audible.com ``` User-agent: sitebot disallow: / ``` (Tells Sitebot Robot to stay out of the website) ``` User-agent: * Disallow: /mycart Disallow: /ajaxcart Disallow: /create-account Disallow: /acc-merge Disallow: /acc-merge6for6 etc.. ``` (Tells all other robots not to enter specific directories listed) Learn more here [http://www.robotstxt.org/robotstxt.html](http://www.robotstxt.org/robotstxt.html)
-
I would appreciate help in understanding what the following robots.txt files are doing.
-
-
Hey Larry, You should be able to put the URL directly into the browser and see the file: http://www.example.com/robots.txt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem on my site
Hello, thank you for guiding me
Competitive Research | | kaajshop 0
I had a problem with the speed of the site. I do not know how to solve this problem
this page:https://kaajshop.com/cat/700 -
What tool can I use to find the top traffic-driving keyword for a batch of pages from multiple sites?
I thought I might be able to use Ahref's Batch Analysis for this, but that just gives stats on backlinks. I have a several lists of thousands of pages scraped for particular sets of keywords, but what I need is some way to automate fetching the biggest keyword that each page ranks for - biggest being the keyword that is estimated to drive the most organic traffic. Is there a tool out there that does something like this?
Competitive Research | | helenlorettahasan0 -
Why do some sites perform great but show no data on the tool bar?
Hey everyone, Been a paid member for 3 days and this is my first time posting 🙂 I have a quick question about competitive analysis with the MozBar. I do not want to point out the exact website (unless that is something cool to do on this forum), but there is a site ranking for a single keyword in the local 7 pack in slot one, which is also slot one for the SERP entirely. When I go to look at the page in more detail, the MozBar is showing no DA, PA, anything at all, and the Open Site Explorer is showing no links internal or external. I know that the page has to have at least some of these elements populated in order to be ranking for a term this competitive, so why is the bar not showing anything? The bar works GREAT on other sites below this one. Thanks for any help!
Competitive Research | | tonyflorespsa0 -
How can stale, poor quality sites rank above fresh, high quality sites?
Dear Community, How is it that a Google search for temperature sensors shows the second result as site **A **(http://goo.gl/zRS1L) in a higher positionthan site B (http://goo.gl/3Xy2O) which is on the 2nd page of the SERPs. Site A blatantly does not follow Google's webmaster guidelines and recommendations for design, content or code. Site B scores an 'A' for on page analysis using SEOmoz, it also has a much higher number of root domains in and Moz rank, we have checked that site B has only quality inbound links, and anchor text is natural, it follows Google's guidelines for content and code diligently, and it is on an older domain. We are satisfied that site B has not been penalised by Google, e.g Penguin algorithm updates. We have also done a website verification with Google. Site B has fresher content compared to site A. Site B also has a sitemap yet site A doesn't. We're satisfied that we can rule out the following Algorithm Factors: Penguin algorithm updates Google Notified penalties Dodgy Inbound links Dodgy Anchor text Website verification with Google Fresh vs. static content Sitemap present Server Speed Server location Domain age and about 170 other algorithm factors Does anyone have any ideas as to why site A is ranking higher than site B in Google.co.uk? Thanks in advance. CD.
Competitive Research | | chichesterdesign0 -
Determining why an established competitor's rankings have bombed - What's the best way to go about it?
I arrived at work this morning to find my weekly SEOmoz ranking report for a main competitor waiting in my inbox. 90% of the their rankings have tanked in the last day or so by an average of 3 pages - most down from page 1 or 2 where they had been sitting pretty for ages. I'm not in a state of (total) euphoria about this because a) you should be humble enough not to gloat at your enemy's demise, and b) I need to find out what they did wrong so that I don't make the same mistake, too. **What is your first suggested port of call to determine where my (vanquished) foe has gone wrong? How much can I find out? ** I do know one thing - with OSE I can see they've used dodgy blogging services but this, to my mind, would have been jumped upon by Google last year. No? Thanks guys
Competitive Research | | Martin_S0 -
I don't get it
Hi...I'm new here....not a professional SEO at all....just been teaching myself as much as I can about SEO on my own as I'm a small business owner desperately trying to make something out of herself. Could use your thoughts on this...please forgive me if this isn't your usual type of question you get here: so...I've been going through all the suggestions for on page optimisation and researching competitors links given through this lovely SEO Moz pro account . One competitor in particular....has spammed A LOT of sites...I think she's hired some company to do it for her. There's article submissions that don't even sound like they are written by someone who speaks English and very low quality sites for back links including some with adult content...however she's got thousands of these links so she ranks extremely high on all keywords. All her pages have very high keyword density and could be accused of keyword stuffing big time. I put her website through the SEOmoz grader....she's got a C and I've got an A. She's completely catered her site to Google not the customer and its obvious. But other competitors in my product have started doing the same thing as this gal and low and behold their sites are popping up in google searches also. Beginning to feel frustrated with my hours of efforts and wondering how I can compete with people like this when I'm trying to be a good girl with Google and focus on creating a great site. My hits per day are increasing slowly, my Alexa rank (hoping this matters) is improving rapidly and you can actually find me on the google search when I couldn't before (I'm page 10 now yay) so I don't feel like a total failure but still am wondering if hitting page 1 for my keywords will happen this lifetime. Why does Google seem to reward people who go against all these countless books and resources on SEO I'm reading? Could use any thoughts/suggestions you might have on this matter. Thanks for your help. x
Competitive Research | | ldnwickless0 -
Is There Video Instruction on How To Use SEOMoz and Open Site Explorer?
Hi I have just signed up to use this software and would like to know how to use it properly. Is there a video tutorial on how to get the most out of the tools? I want to get the competition's backlinks but for some reason it's not openiing the page. I also want to know how the backlinks are assessed to find the real quality ones that I can link to. How are the backlinks assesed and what standards are used to asses backlink quality? Thank you Sunil.
Competitive Research | | sunilmuse0 -
Image only site on top of Google
Hi Everyone, I'm trying to rank in Google for 'Hid xenon' in the netherlands, but there is one site above all results: http://bit.ly/qlsjne As you can see the site almost has no backlinks, and has not a single word in it's content, all images. it's a keyword only domain, and that's probably the only reason why it's ranking that high, but that means then that i can never get higher then him in Google because of it's domainname? Even when it's such a shitty site? Thank you, regards yannick
Competitive Research | | iwebdevnl0