How do you stop Moz crawling a page?

[Seaward-Group]

Hello,

I have a contact form which generates thousands of duplicate crawl errors. I'm going to use  to block Google indexing these pages. Will this also block MOZ from crawling these pages and displaying the error?

Thanks!

[signcreator]

this is all well and good and I am able to do these, but how do I keep Moz from crawling an index.php file. our site is http://4signs.com  no index file there at all so I'm not sure why it would be crawled.

thoughts?

[Peterli]

Hi guys,

Awesome discussion so far Yes, Chris is correct in that using noindex as a way to block Moz is not a effective way to do it.  Since our tool is not a typical indexer (such as Google), we don't have some of the behavior of a normal spider.  Instead, Roger is very good at rooting out issues that other crawlers might not notice.   One thing Roger is also good at is obeying robots.txt.... you know him being a robot and all

You can find more information about our friend here:

http://moz.com/learn/seo/robotstxt http://moz.com/help/pro/rogerbot-crawler

So if you are looking to block it from looking at a page without making content changes to your code, I would definitely look into using robots.txt.  You can even use a user-agent specific directive to make sure you don't end up telling other robots/spiders to do the same thing.

I hope that helps!  Please let us know if you have questions

Peter
Moz Help Team.

[Chris.Menke]

On http://moz.com/help/pro/rogerbot-crawler Moz gives an answer to the question "We are still seeing duplicate content on Moz even though we have marked those pages as "noindex, follow. Any idea why?

Moz is not a search engine index, it uses a crawler. If those pages are not blocked by the robots.txt file, then Moz will crawl them. They ignore the noindex tag because they don't index anything. Search engines will honor the noindex tag and not index a page if you specify with the robots meta tag. However, to remove pages from the crawl, disallow them in the robots.txt or metarobots.

Their answer is not exactly clear, but according to it, no, a meta noindex  will not block rogerbot from crawling your page.

[GPainter]

Hi Gary,

You may find the following link helpful - http://moz.com/learn/seo/robotstxt on top of this you can read how to stop the moz bot here - http://moz.com/help/pro/rogerbot-crawler

If you have blocked bots from your page this will include the Mozbot. Hope this helps.

[AndreVanKets]

Yes it will.