Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website Issues - Duplicate Content
Hello, I'm fairly new to using Moz and I logged on this morning to find Issues have been found in one of the websites - 22 High Priority and 44 Medium. I know it's due to duplicate content in the blog, but i can't figure out what is duplicated? I've only recently come on board this website so I don't know if the content has been plagiarised or what? The link to the site is here: delacyspa.co.uk Any help would be appreciated. Thanks zFxQmmd
Moz Pro | | Cowbang0 -
Internal Links Data on OSE and poor ranking despite excellent metrics
Hi folks, I use OSE a lot for comparing where we stand against our competitors and I'm always surprised to see most of our competitors have loads more internal links than we do. For example, comparing 'www.kensyard.co.uk' with 'www.sterlingbuild.co.uk', we are better on every Root Domain Metric except for the Total Links metric and this is down to a much smaller number of internal links. They are ranking better than us in Google so I'm wondering if there is something wrong with the way our site is coded / structured that means Google and OSE are not picking up our internal link network properly? We have over nine times as many external root domains linking to our site than theirs, which is surely a far more valuable metric than internal links right? Or is there an issue with our internal link structure that is confusing Google and causing us to rank lower? Many thanks for any advice, Luke
Moz Pro | | LukeyB300 -
Broad Website - What keyword to target
Hello, I have a Brand name Roclay; and my domain is roclay.com . It is technology blog and we cover literally all technologies etc and every article is unique with it's own keyword. Please help me as I am facing issue of SEO for many months now. What keyword should I focus on and how ? Moreover what should I setup in moz.com for keyword of my website when every single article is different. Regards
Moz Pro | | roclay0 -
Do the SEOmoz Campaign Reports follow Robots.txt?
Hello, Do the SEOmoz Campaign Reports (that track errors and warnings for a website) follow rules I write in the robots.txt file? I've done all that I can to fix the legitimate errors with my website, as reported by the fabulous SEOmoz tools. I want to clean up my pages indexed with the search engines so I've written a few rules to exclude content from Wordpress tag URLs for instance. Will my campaign report errors and warnings also drop as a result of this?
Moz Pro | | Flexcin0 -
OSE lists dead links
Going over the link profile of a competitor who gets 5x the traffic we do.... of course frustrated that the majority of their links are spam blogs (full of words but don't communicate anything) and forum profiles. Thanks Google for telling me what not to do, then rewarding my competitor for doing it shamelessly. Question regarding sites listed by Open Site Explorer as linking to said competitor, but that don't even load when I visit their url. Some go to a godaddy parked page, like the domain name expired long ago. Is this simply a limitation of OSE, and can I assume Google has indexed differently and therefore awarding no link juice from these urls?
Moz Pro | | jotham20 -
I'm not getting the csv emailed in OSE? Anyone else?
The new version of the tool does not allow me to download the data via csv immediately. It says it will email in 5-10 minutes and never comes? Anyone else having this problem? Cheers
Moz Pro | | josey0 -
In OSE is there a way to see the total number or links coming from each domain?
Open site explorer will only show up to to 25 links per domain but in order to fully understand the link profile for the site I'm working on, I need to know how many links are coming from each domain - when I download the linking domains report, it doesn't contain this information. Is there a way to access it?
Moz Pro | | jabbate0