Standard Syntax in robots.txt doesn't prevent Moz bot from crawling
-
A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/
To resolve this we have set up a disallow statement in the robots.txt file that says
Disallow: /page/For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?
-
Thanks, Tawny,
If you look at Duplicate titles, check the first one (https://needquest.com/place_tag/autism-spectrum-disorder/). All the URLs with a duplicate title have /page/ in them. I will suggest they move the Allow statement and see if that helps.
-
I'm not seeing that URL coming up with Duplicate Title or Duplicate Content issues — when I search by that URL I see no Content issues at that URL. I do see that URL in the All Crawled Pages section, but I can't find it bringing up Content issues in the app.
That said, I took a look at your robots.txt file, and I think this could be a result of having an Allow command before the rest of the Disallow commands. I think possibly if you put that Allow command at the end of the block of Disallow commands, rogerbot would see the disallow for /page/ and stop crawling those URLs.
If you're still running into trouble, I would suggest writing in to us at help@moz.com so we can take a closer look at the Campaign and what could be going on there.
-
Any reason the Disallow: /page/ isn't preventing URLs like
https://needquest.com/place_tag/autism-spectrum-disorder**/page/**4/
from generating duplicate descriptions and title errors in our site crawl? It was my hope that those pages wouldn't be crawled at all. -
Sorry, Tawny ... I did go back and correct y question. We did apply Disallow: /page/ to address this issue. The /place_tag/ is found in many pages we DO want to crawl and index ... and we only want here to disallow those page 2, page 3, page 4, etc. pages.
(We also disallowed /tag/, /category/, and a few other common issues that generate false positives in the site crawl.)
-
Hey there!
Tawny from Moz's Help Team here.
Adding a disallow directive for /tag/ won't help with the example URL you've provided — that URL doesn't have /tag/ in the URL pathway. To block us from seeing content like that URL you listed, you'd need a disallow directive for /place_tag/.
If you include that disallow directive, that should stop us from seeing duplicate content on pages with /place_tag/ in the URL.
Hope that helps! If you've still got questions, feel free to shoot us a note over at help@moz.com and we'll do our best to sort things out with you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
SSL - green padlock but Moz say there's an 804 error?
Hi, my site has a green padlock and no SSL errors but Moz are reporting an 804 error. I use CloudFlare with fairly complex settings. I've read this thread but it's quite old and I don't understand which parts of it are still valid. I'd love to know whether this can be sorted before I spend hours setting up Moz's features as if they can't crawl my site then I would obviously need to cancel my subscription. Thanks
Getting Started | | Barn2Plugins0 -
Setting up Moz Pro campaign for a subfolder
Hi, I'm just setting up my first campaign and would like some advice please. I have a website and also within it a a number of pages in a sub folder. These are technically a different business to the main website and I would like to track them separately in Moz so they can have their own keyword lists and competitors etc. How do I do this? Do I set up two separate campaigns www.site.com
Getting Started | | craigramsay
www.site.com/subfolder I understand the first one might still report on the subfolder but I guess I could just ignore this. Thanks0 -
What is the best use for Moz tools specially keyword difficulty for startup ?
Hello, I'm so new in Moz and SEO world and i just started my website, a WordPress blog, I'm in a content creation period and i want to make it right from the beginning but I'm confused about how to use Moz tools in this period because i don't have content or traffic so no analytic as i think, so What is the best use of Moz tools in this period? About keyword difficulty tool i think this is the most tool i will use in the beginning, how i choose which keywords to use from my keywords list, in this time I'm depending on the on page SEO only, no backlinks no social engagements, which keywords to use to appear fast in search engines for a startup? less than "% difficulty " or between, I"m new in this word Please Moz and SEO experts give me a hand here. Note: I'm using Medium Moz pro plan.
Getting Started | | Romekio1 -
Can I upgrade my moz subscription from large to premium without getting my campaigns being deleted?Do i need to wait till the end of the month to resubscribe?
I'm planning to take a large subscription of moz..but i may need a premium account in the future.Can I upgrade it in the middle of my current subscription or will i have to wait till my subscription gets over? Also, I'm planning to take only one month currently but i'll renew it once the month completes so can i continue the same account without my campaigns being deleted?
Getting Started | | kdcdmp0 -
I can't export reports in pdf
Hi,I try to export pdf report for my client but this function doesn't work. I tried in different browsers and on different machines - still doesn't work. What am I doing wrong? Thanks in advance,JJ
Getting Started | | jjtech0 -
Need to Add Moz code?
Do i need to add tracking code in my website, for Moz tool to analyze my website. Please someone let me know.
Getting Started | | TOBOC0 -
How do you "Moz Crawl" a website? Newbie...
Hi everyone; I've used Screaming Frog in the past and it's simple: you enter the URL to the box and click "start" and.. voila: as the button says, the crawling starts. I've had the Pro version of Moz for a while now and haven't really 'done' anything with it. I'd like to crawl a website and thought it would be as easy as it's always been with Screaming Frog... but, for some reason, I can't find the 'way' to do it. I find it really frustrating especially cause I feel like an idiot going around in circles thinking I'm missing something really obvious... Until I realised the only solution was to ask here! So... how in the world do you crawl a website using Moz tools? (Pro version) Thanks!
Getting Started | | patrihernandez1