Website blocked by Robots.txt in OSE
-
When viewing my client's website in OSE under the Top Pages tab, it shows that ALL pages are blocked by Robots.txt. This is extremely concerning because Google Webmaster Tools is showing me that all pages are indexed and OK. No crawl errors, no messages, no nothing. I did a "site:website.com" in Google and all of the pages of the website returned.
Any thoughts? Where is OSE picking up this signal? I cannot find a blocked robots tag in the code or anything.
-
No worries - glad to help!
-
Thanks for responding - I did, and I noticed that we are blocking a bunch of other spiders including the spider that crawls for OSE. So, that explains why they cannot retrieve the data.
Again, thanks.
-
Have you looked at your robots.txt file to see if you are blocking specific bots? Visit yoursite.com/robots.txt and check whether you have something like this:
User-agent: [example]
Disallow: /But you may have something else to specify that Googlebot is allowed to crawl the site:
User-agent: googlebot
Allow: /
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My website was at the top of Google search for some years... suddenly I almost can't reach first page! Moz ranks my website better than the competitors... what might be going one? Could anybody help me out? Thanks!
Hello Guys! My website was at the top of Google search for some years... suddenly I almost can't reach first page! Moz ranks my website better than the competitors... what might be going one? Could anybody help me out? Moz rank us grade A... the competitors B or C .. I think we have better back links than they do... Would you need any kind of data or report to help me here? Thanks!
Moz Pro | | wesleyms0 -
A competitor's SEO firm is building spammy links to my website.
A competitor's SEO firm is building spammy links to my website. I know the disavow link process, but even this can be time consuming. Is there a better way to protect my website? Any and all ideas appreciated.... Are there any cool Moz tools that can help me manage this assault? Sorry, just a follow up from me... It appears that many of the spammy links (referrals) I see in Google Analytics are not present in Google WebMaster Tools. Does this mean that the spammy ones that I'm concerned about have already been discounted by Google, or are the spammy links no-follow? Any input on this would also be appreciated.
Moz Pro | | BVREID0 -
Website not being indexed by Google - does seomoz have a index checker?
i have been to a couple of index checking websites that are saying my website is not being indexed, my developer and seo team have been making a lot of changes recently and I think it is because of that. Also my first byte grade is D. Can I check here on seomoz if my site is being indexed by Google here in the UK?
Moz Pro | | Malx0 -
Discrepancy in OSE data
When I look up a domain in OSE, it shows that there are 3K+ links from 349 domains. However, when I download the CSV, I only see around 1,500 links. Why the discrepancy?
Moz Pro | | inhouseseo0 -
Sudden, drastic drop in # of links and root domains on website?
Noticed that my links and root domain numbers had dropped by ~40%, seemingly overnight. Any idea what could have contributed to this (Toolbar update???)?
Moz Pro | | wlefevre0 -
How can I Pull OSE Data for Multiple URL's at once
I'm putting together a link prospecting csv (very basic/simple). I'm doing my own manual hunting for link prospects and compiling them in a list in that excel doc. Once I'm done with that, I want to pull OSE data on a larger scale (MozRank, PA, DA, etc.). I know Niel Bosma's SEO tools for Excel exists, but I have a Mac, and it's not available for that. And I can't really pay for any of the big tools right now (ie BuzzStream). Does anybody know of a good tool or way of going about pulling this data in a way that will save time? As opposed to pulling data for each URL one by one. ANY tips would be GREATLY appreciated.
Moz Pro | | MichaelWeisbaum0 -
Does the SEOMoz weekly crawl that highlights no meta description tag, take into account if there is a meta robots noindex,follow tag on the pages it indicates the missing meta descriptions?
The weekly crawl website report is telling me that there are pages that have missing meta description tags, yet I've implemented meta robots tags to 'noindex, follow' those pages which are visible in those page source files. As far as Google Is concerned, surely this then won't be a problem since it is being instructed NOT to consider these specific pages for indexing. I am assuming that the weekly SEOmoz website crawl is simply throwing the missing meta description crawl findings into its report without itself observing that the particluar URL references contain the meta robots 'noindex,follow' tag ???? Appreciate if you can clairfy if this is the case. It would help me understand that (at least in terms of my efforts towards Google) your own crawl doesn't observe the meta robots tag instruction, hence the resultant report's flagging the discrepancy.
Moz Pro | | callassist0 -
Why is the SEOmoz crawler crawling the old version of our website?
Hello, I'm a new SEOmoz member. On Dec. 2nd, after completely redesigning our website, we migrated to a new hosting company by switching our DNS to the new server. The vast majority of the URLs have changed and we configured redirects of the old URLs to the new ones. Although, this task is not completed yet. After the migration, I created an account on SEOmoz to be able to track our progress and find the issues to fix to optimize our SEO. For some reason, in the SEOmoz reports it is the old URLs that show up. Unless the crawler does not actually crawl the pages and only uses the indexed pages to generate its report, I don't understand how could this possible. Anyone has a clue? When will the new URLs be indexed by SEOmoz and the major search engines? Thanks for your help!
Moz Pro | | Gestisoft-Qc0