Moz "Crawl Diagnostics" doesn't respect robots.txt
-
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like:
- Duplicate content
- Overly dynamic URLs
- Duplicate Page Titles
The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored):Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/Many thanks for any info on this issue.
-
Hi Si, has this issue been resolved?
-
Hey Si,
Thanks for writing in. It doesn't seem that we are having an overarching issue with our crawler ignoring robots.txt files so I did some research in Google Webmaster Tools and it looks like most crawlers require an asterisk in the disallow directive to recognize that all pages of a dynamic URL are being disallowed. If you look in the "Pattern Matching" section of this resource here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449, that should give you more information about setting up the robots.txt with the correct disallow directives to block those pages.
If you add in the astrisk to the disallow directive and you are still seeing these pages crawled, it would help if you sent in an email with your campaign information to our support desk at help@moz.com so we can have our engineers look into this more directly.
I hope this helps.
Chiaryn
-
If you have an "index,(no)follow" meta on those pages I think they will be crawled even though you have them blocked in robots.txt. So by adding "noindex" on those pages it might work as you want it to.
-
Is the / actually in the URL at that spot? Or is your link like http://www.example.com/abcd?p=147
If you give an example full URL that includes one of your blocked dynamic URLs we can take a better look. If your robots is setup correctly, it shouldn't find that stuff but give us more info if you're able.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Did Moz Bar change to have no Keyword capabilities?
I'm trying to use it during SEO training over here and the KW button goes to a "Get Page Optimization Score with MozBar Premium - Try Free", with a link that takes you to: https://hsinfo.moz.com/mozpro/mozbar/lander?utm_medium=cpc&utm_source=google&utm_campaign=Brand | NA&utm_adgroup=Brand - MozBar&utm_term=mozbar&gclid=CjwKCAjwxOCRBhA8EiwA0X8hi4Tx1YaOVwzWZaCGmzmYdBO4JEON8YlRMw52stp2AyfEBbH4uWDnARoCum0QAvD_BwE? Didn't this used to accept keywords and allow keyword checking? My training materials have the KW button behaving like this: [1] Do Keyword Research in MozBar.
Moz Bar | | EricaJorgensen
1. Click the icon with KW and a magnifying glass.
2. Enter a term related to your subject. For example, "cyber security".
There is a section telling you the keyword score, the relevancy to your page, and giving you optimizations that you can make to the page regarding this term.... Thus, I feel sure KW had useful and free features and not a button for a trial and paid Moz Bar Premium account. What is the pricing for this feature now? Or am I missing something? Thanks,
Tracy!0 -
"New" issues not previously found being shown?
I'm not sure what logic Moz is using for its reporting of Site Crawl issues, but it appears to be pretty flawed (unless I'm missing something, which is possible). I've got a client site that has been in Moz for about 6 months now. Every time the crawler runs, the same number of pages are reported as having been crawled. However I'm consistently getting "New Issues" reported that should have been reported during previous crawls. Example: A redirect chain was reported several month ago. The referring URL was the homepage of the website, and we tracked it down to an old link in the header. This was fixed, marked as resolved, and the issue was not shown on the next crawl. Several weeks later, the same issue was reported for a different page on the website - a page which has existed since 2014 and was already crawled many times. Again, we fixed. Fast-forward to the report that just ran on 12/1 and we have the same issue reported, for a different page, which has also existed for years and has been previously crawled. It's very hard to explain to a client "this item you are seeing has been resolved", only to have it continually crop back up in future reports. Note this is not limited to redirect chains - that's just an example. I'm seeing this for other items such as missing canonicals, duplicate titles, etc.
Moz Bar | | RucksackDigital0 -
What data we don't get from link explorer that we can get if we add a campaign?
I was wondering what's the difference between campaign data and link explorer data, both in pro version of moz? What are the features we get by adding campaign that we don't get via link explorer?
Moz Bar | | HuptechWebseo0 -
Why is Moz Crawling More Pages Than My Site Actually Has?
Hi I have a site that only has 5k pages but Moz has crawled 50K pages on the site when I initiated the site crawl. I don't exactly know why Moz is reporting me back so many pages but I was wondering why this is and if any of you out in the Moz community know anything about this. Thanks
Moz Bar | | drewstorys0 -
Lack of UK Keyword Volume Data In Moz
Is it just me or is there a considerable lack of keyword Volume data for UK Google search terms on Moz? I have 53 keywords and not a single one has any keyword volume data - these are not obscure terms and include the following as examples... leather satchel, leather laptop bag, satchels, leather bag, leather backpack, leather school satchel. Without this information aren't a lot of the services offered by Moz rather academic as it is impossible to know which terms are really worth targeting. What is the solution? I could use US data and hope it is similar but this seems close to a deal breaker for UK subscribers.
Moz Bar | | MrFrisbee0 -
Is there a way to track mobile rankings vs desktop rankings in Moz?
With the new release of Google's mobile algorithm we want to start tracking keywords mobile vs desktop. Any suggestions?
Moz Bar | | TicketCity3 -
Why can't On-Page Grader grade any Hilton hotel URLs?
I'm receiving the "Sorry, but that URL is inaccessible." for every hilton hotel webpage I check when using On-Page Grader. Is Hilton blocking Moz's On-Page Grader or is something else going on? Here are a few "inaccessible URLs" from different brands within Hilton's portfolio: http://doubletree3.hilton.com/en/hotels/new-york/doubletree-by-hilton-hotel-metropolitan-new-york-city-NYCDTDT/index.html http://home2suites3.hilton.com/en/hotels/tennessee/home2-suites-by-hilton-nashville-vanderbilt-tn-BNAHTHT/index.html http://hamptoninn3.hilton.com/en/hotels/florida/hampton-inn-and-suites-destin-DSINEHX/index.html http://hiltongardeninn3.hilton.com/en/hotels/georgia/hilton-garden-inn-atlanta-downtown-ATLDOGI/index.html Thanks in advance.
Moz Bar | | Just-Me0 -
On Page Grade - Moz Analytics
When I go to search overview I can see my on page optimization grades, but when I try to use the on page grade optimizer for different keywords and url's I get the error. "The URL you entered does not appear to be returning a page successfully. Please make sure that you've entered the URL of valid, working page." However, I know it is a valid url and I get the error even if I use the same url and same keyword I can see the grade for from the search overview. Also, if I try to regrade the page nothing happens and where my previous grade and analysis was located I now have nothing. Any information about why this is happening? Thanks for the help.
Moz Bar | | ZeroWing0