Moz "Crawl Diagnostics" doesn't respect robots.txt
-
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like:
- Duplicate content
- Overly dynamic URLs
- Duplicate Page Titles
The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored):Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/Many thanks for any info on this issue.
-
Hi Si, has this issue been resolved?
-
Hey Si,
Thanks for writing in. It doesn't seem that we are having an overarching issue with our crawler ignoring robots.txt files so I did some research in Google Webmaster Tools and it looks like most crawlers require an asterisk in the disallow directive to recognize that all pages of a dynamic URL are being disallowed. If you look in the "Pattern Matching" section of this resource here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449, that should give you more information about setting up the robots.txt with the correct disallow directives to block those pages.
If you add in the astrisk to the disallow directive and you are still seeing these pages crawled, it would help if you sent in an email with your campaign information to our support desk at help@moz.com so we can have our engineers look into this more directly.
I hope this helps.
Chiaryn
-
If you have an "index,(no)follow" meta on those pages I think they will be crawled even though you have them blocked in robots.txt. So by adding "noindex" on those pages it might work as you want it to.
-
Is the / actually in the URL at that spot? Or is your link like http://www.example.com/abcd?p=147
If you give an example full URL that includes one of your blocked dynamic URLs we can take a better look. If your robots is setup correctly, it shouldn't find that stuff but give us more info if you're able.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How accurate is Moz Rank Tracker tool? It's showing different results than a Google incognito search.
I have a keyword/url combo with Moz Rank Tracker showing 3 spots above what a Google Incognito search showed. I performed my Google Incognito search based on these suggestions: https://moz.com/community/q/best-and-easiest-google-depersonalization-method Is the Moz Rank Tracker tool off?
Moz Bar | | chiefmoz1 -
What does it mean when Moz KW explorer returns a negative keyword difficulty score (eg -2)?
I recently did a keyowrd difficulty lookup in Moz keyword explorer as usual and for the first time a saw several (out of 100s) of negative keyowrd scores (generally -2). Is this a bug, what does it mean?
Moz Bar | | FlagshipCons1 -
Is the Moz on-page grader going to start grading mobile-first as Google does?
I wonder whether this has been taken into account yet or there are any plans to in future.
Moz Bar | | mybuilder1 -
MOZ Staff: Timeline for supporting SNI?
We have moved our blog to Amazon Web Services, and our website is soon to follow. For better or worse, AWS uses SNI, which MOZ doesn't currently support. Here are some recent forum posts about it: https://moz.com/community/q/804-server-error-crawling-https https://moz.com/community/q/804-https-ssl-error This makes MOZ much, much less useful to me. MOZ staff, you have a timeline for when you'll implement support for SNI?
Moz Bar | | Atomic-Object2 -
My crawl report only shows 1 link
Hello, I've tried a crawl for the site www.doctify.co.uk and it's only returned 1 link in the report which is the homepage. Do you know what the issue could be? Thanks, Nina
Moz Bar | | Global_Blue0 -
Does anyone else have issues with Moz's keyword search volume tool for Google's search engine?
It will show the search volume for Bing even when Google is selected. Then, if you select Bing, you'll get the same data as it shows for when you selected "google". So basically, this tool does not work for Google's search engine. Or it is most likely not a reliable way to perform keyword research. Anyone else notice this? Does Moz even offer a way to submit a support ticket to get this fixed?
Moz Bar | | ShokIdeaGroup1 -
I requested a new crawl, this was done but my dashboard only shows the crawl done last week?
We recently moved our old website to a new CMS and structure. there have been some configuration errors and I needed to make some changes with things like canonical url's etc. However I need to check if these changes have made a difference and requested a new crawl through the crawl test page. I was emailed each time that a new crawl had been done but my reporting and dashboards still only show data from the last scheduled crawl. Regards Chris
Moz Bar | | LRQA-Marketing0 -
Is Manual Crawl Test option available now to Pro Users?
Hi all, I have worked on my Crawl Issues and want to see how many still exist. Earlier I was using Manual Crawl Test. However, now I don't see this tool in Moz Account. Please suggest. Thanks
Moz Bar | | chandman0