Crawl Diagnostics Summary Problem
-
We added our website a Robots.txt file and there are pages blocked by robots.txt. Crawl Diagnostics Summary page shows there is no page blocked by Robots.txt. Why?
-
Hey there,
Thanks for the question. How you have your robots.txt set is actually preventing all bots from even touching on those pages, not just the engines.
If you had a directive allowing RogerBot access to those pages it would be able to touch on them and register that they are blocked from the Search Engines in the robots.txt.
Since our crawler strictly adheres to the robots.txt file you won't have anything populated there.I hope that makes sense. Feel free to reach out if you need more information.
Cheers,
Joel. -
Thanks Federico,
Can we use meta robots noindex and robots.txt together?
-
I am guessing here, but Moz crawler does not respect your robots.txt file. Instead, if you want pages not to be crawled, try using the meta robots noindex for a change and see what happens.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SEOmoz crawler not crawling my site
We set up a new campaign in SEOmoz on Friday. It is my understanding that the preliminary crawl can cover up to 250 and this has been our experience in the past. However, the preliminary crawl only went through 2 pages. This is a larger eCommerce site with many pages. Any ideas why more pages weren't crawled? We set up the campaign to track at the root domain level.
Moz Pro | | IMM0 -
Crawl Diagnostics - Crawling way more pages than my site has?
Hello all, I'm fairly new here, more of a paid search guy dabbling in SEO on the side. I have a client that I have in SEOMoz and the Crawl Diagnostics report is showing 10,000+ pages crawled and I think the site has at most 800 pages (e-commerce site using freewebstore.org as the platform). Any reasons this would be happening?
Moz Pro | | LodestoneGen0 -
Is anyone having problems with rank tracker
Hi for some reason i am finding semoz site very slow and when i have tried to use rank tracker it is not working. is anyone else finding this
Moz Pro | | ClaireH-1848860 -
Crawler has stopped crawling my website
Hi all, I recently changed my site www.ipbskinning.com to ipbskinning.com. For some reason my weekly scan states that no pages have been crawled. I tried making a new campaign with the sub-domain ipbskinning.com but still the starter crawl returns nothing? Am I doing something wrong? :S Thanks all!
Moz Pro | | pezza34340 -
Did anyone else see "Rel Canonical" drop to zero after their latest SEOmoz crawl?
In the Crawl Diagnostics section of the SEOmoz reports, we get errors in red, warnings in yellow, and notices in blue. After my latest crawl, I saw the "Rel Canonical" part go from about 300 down to 0. Obviously, this isn't right, so I'm wondering if this is a bug that everyone is experiencing. U9W5I
Moz Pro | | UnderRugSwept0 -
Seo moz crawl is not updating
When I check our seo campaign I can see that the report was not updated. It still show that the next crawl is Nov 1 but it is already Nov 3.
Moz Pro | | shebinhassan0 -
How to handle crawl diagnostic errors for the same url. /products & /products/
I have copied on of the errors out of the crawl diagnostics report. Both /products and /products/ are returning an error, and both have pretty good domain authority so I feel like its hurting my site that these show up this way. Both urls create the same page, should I just setup a 301 on the /products with no slash or will that cause more harm... I am using the MODx cms system and that could have something to do with it. | Products | Datalight http://www.datalight.com/products 1 37 5 Products | Datalight http://www.datalight.com/products/ | 1 | 30 | 1 |
Moz Pro | | tjsherrill0