Moz "Crawl Diagnostics" doesn't respect robots.txt
-
Hello, I've just had a new website crawled by the Moz bot. It's come back with thousands of errors saying things like:
- Duplicate content
- Overly dynamic URLs
- Duplicate Page Titles
The duplicate content & URLs it's found are all blocked in the robots.txt so why am I seeing these errors?
Here's an example of some of the robots.txt that blocks things like dynamic URLs and directories (which Moz bot ignored):Disallow: /?mode=
Disallow: /?limit=
Disallow: /?dir=
Disallow: /?p=*&
Disallow: /?SID=
Disallow: /reviews/
Disallow: /home/Many thanks for any info on this issue.
-
Hi Si, has this issue been resolved?
-
Hey Si,
Thanks for writing in. It doesn't seem that we are having an overarching issue with our crawler ignoring robots.txt files so I did some research in Google Webmaster Tools and it looks like most crawlers require an asterisk in the disallow directive to recognize that all pages of a dynamic URL are being disallowed. If you look in the "Pattern Matching" section of this resource here: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449, that should give you more information about setting up the robots.txt with the correct disallow directives to block those pages.
If you add in the astrisk to the disallow directive and you are still seeing these pages crawled, it would help if you sent in an email with your campaign information to our support desk at help@moz.com so we can have our engineers look into this more directly.
I hope this helps.
Chiaryn
-
If you have an "index,(no)follow" meta on those pages I think they will be crawled even though you have them blocked in robots.txt. So by adding "noindex" on those pages it might work as you want it to.
-
Is the / actually in the URL at that spot? Or is your link like http://www.example.com/abcd?p=147
If you give an example full URL that includes one of your blocked dynamic URLs we can take a better look. If your robots is setup correctly, it shouldn't find that stuff but give us more info if you're able.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links on my website do not get highlighted by Moz bar in Chrome
I've used the Moz bar for many years to quickly figure out if a link is followed or no-followed. I recently used a Wordpress plugin to build lists of subjects and excerpts of pages with appropriate links. An example is at https://www.chicagotraveler.com/chicago-parks/ This is the main Parks section page. Below the map is a set of links and descriptions of each park in this section of the site. The problem is that when I use the Moz bar to look at this page, the links are not highlighted no matter which settings I click on. Followed, No-followed, External or Internal. I've looked at the code and while there is a bit of css nearby the links, they look fairly normal. Does this mean that moz thinks there is something wrong. Do you think google will also ignore these links? Should I scrap the plugin and build and maintain these lists manually?
Moz Bar | | EdKim0 -
Why do my Moz duplicate content results show me pages with no noticeably similar content?
Sometimes the "Pages with Duplicate Content" results under Content Issues show pages that, from what I'm able to see or otherwise test, have no duplicate content, save for the same navigation that exists on all of my pages. For example, a recent issue said that the following pages had duplicate content:
Moz Bar | | rickmic
https://freezerworks.com/index.php/html/slider-overlay
https://freezerworks.com/index.php/ufaqs/what-do-i-get-with-my-purchase-of-freezerworks
https://freezerworks.com/index.php/videos/fda-and-freezerworks-2
https://freezerworks.com/index.php/lims-testing-module Even a side-by-side of the page source in a text comparison tool shows nothing but navigation and scripts used in every page. Am I not seeing something?2 -
What data we don't get from link explorer that we can get if we add a campaign?
I was wondering what's the difference between campaign data and link explorer data, both in pro version of moz? What are the features we get by adding campaign that we don't get via link explorer?
Moz Bar | | HuptechWebseo0 -
Moz is finding phantom pages
I suddenly have 4xx errors in my crawl diagnostics because pages with “/%3C/div” added to the end of the URL that are linked from the normal page can't be found. I didn't create the pages, and they don't exist, but Moz thinks that they do. I went back through to see if any changes in WordPress, theme or plugins versions might be the cause, but this is the only site that I have this issue, so I don't think that is it. Does anyone have an idea what causes this?
Moz Bar | | samuelldrew0 -
The Old Moz Pro
Hello, I found the old moz pro a lot more informative for certain queries. Can I still access this and run a campaignMany Thanks
Moz Bar | | summer3000 -
Conversion tracking within Moz reports?
Hey guys, Quick question and apologies if this has been asked before, but is there an option within MOZ reporting (Pro) that allows you to focus on conversions for your clients? I'm finding rankings to be a very poor indicator of success. I'd much rather focus on conversions, and present this information to the client within my reports. Thanks
Moz Bar | | John_Romaine2 -
I am not able to perform crawl test in moz tools
it is throwing there is some problem in domain when i try testing the crawl test for my domains
Moz Bar | | IBEE-Hosting0 -
On Page Grade - Moz Analytics
When I go to search overview I can see my on page optimization grades, but when I try to use the on page grade optimizer for different keywords and url's I get the error. "The URL you entered does not appear to be returning a page successfully. Please make sure that you've entered the URL of valid, working page." However, I know it is a valid url and I get the error even if I use the same url and same keyword I can see the grade for from the search overview. Also, if I try to regrade the page nothing happens and where my previous grade and analysis was located I now have nothing. Any information about why this is happening? Thanks for the help.
Moz Bar | | ZeroWing0