How do you stop Moz crawling a page?
-
Hello,
I have a contact form which generates thousands of duplicate crawl errors. I'm going to use to block Google indexing these pages. Will this also block MOZ from crawling these pages and displaying the error?
Thanks!
-
this is all well and good and I am able to do these, but how do I keep Moz from crawling an index.php file. our site is http://4signs.com no index file there at all so I'm not sure why it would be crawled.
thoughts?
-
Hi guys,
Awesome discussion so far Yes, Chris is correct in that using noindex as a way to block Moz is not a effective way to do it. Since our tool is not a typical indexer (such as Google), we don't have some of the behavior of a normal spider. Instead, Roger is very good at rooting out issues that other crawlers might not notice. One thing Roger is also good at is obeying robots.txt.... you know him being a robot and all
You can find more information about our friend here:
http://moz.com/learn/seo/robotstxt http://moz.com/help/pro/rogerbot-crawler
So if you are looking to block it from looking at a page without making content changes to your code, I would definitely look into using robots.txt. You can even use a user-agent specific directive to make sure you don't end up telling other robots/spiders to do the same thing.
I hope that helps! Please let us know if you have questions
Peter
Moz Help Team. -
On http://moz.com/help/pro/rogerbot-crawler Moz gives an answer to the question "We are still seeing duplicate content on Moz even though we have marked those pages as "noindex, follow. Any idea why?
Moz is not a search engine index, it uses a crawler. If those pages are not blocked by the robots.txt file, then Moz will crawl them. They ignore the noindex tag because they don't index anything. Search engines will honor the noindex tag and not index a page if you specify with the robots meta tag. However, to remove pages from the crawl, disallow them in the robots.txt or metarobots.
Their answer is not exactly clear, but according to it, no, a meta noindex will not block rogerbot from crawling your page.
-
Hi Gary,
You may find the following link helpful - http://moz.com/learn/seo/robotstxt on top of this you can read how to stop the moz bot here - http://moz.com/help/pro/rogerbot-crawler
If you have blocked bots from your page this will include the Mozbot. Hope this helps.
-
Yes it will.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Current Title is short but Moz show error that Title is Too Long
Hello, The current title of an article is of 30 characters. But in Moz it say "Title Element Too Long". The URL of the article is long. Please let me know why Moz is showing it as "Title Element Too Long"
Moz Bar | | ProcessSEO0 -
On-Page Grader fails?
I have optimized the following page for the norwegian word "Dusjkabinett": https://www.vvskupp.no/Produkter/Baderom/Dusjkabinett The On-Page grader gives the URL an F-grade. BUT as you can see it seems like the tool fails in giving a correct report. See attachment. Can anybody help me understand? pCyEYaWd3
Moz Bar | | moggiew0 -
On Page Grader inconsistent
Why does the on page grader not update it's grades based on the other factors, other then the title tag. i.e. I can have the keyword 'burger' in H1 tags, the URL, ALT attributes, in the body text a couple times and in the meta section of the site and receive an F grade from the on page grader, but as soon as I add the word 'burger' to the title I receive an A grade. Is there a reason why the only factor that has that influence is the title tag, and why is it that a keyword for a page cant get say a B or C grade if it has the other factors covered (i.e. they have a tick next to them) but not the title tag? Cheers Again
Moz Bar | | sharpleaddesign0 -
Moz Temperature Widget
I've just tried to add the Moz temperature widget to my site (https) and it is getting blocked by Google Chrome for an incorrectly configured SSL certificate (apparently!) The widget embeds this page - https://mozcast.com/embed/yesterday/horizontal Which comes up with a security issue when visited in Chrome Do I need to wait for Moz to fix? Kind Regards Liam
Moz Bar | | ZaddleMarketing0 -
How much time should I wait between Crawl Tests?
Hello! I ask because it has happened before (and again this morning) that after doing a crawl test and repairing my site per the errors found in Moz's crawl test it still finds the same error. Even though I fixed them. Typically I do a re-crawl 6 hours after or the next day and I find the same errors. I know they are fixed because a couple of days go by and finally Moz gets it right. I had understood that the crawl test was an "on-demand" crawl of sorts, granted with limit of 2 a day. But it seems that if you re-crawl your site within a day the same results yield? It's frustrating. Is this correct? Thank you!
Moz Bar | | md30 -
Perplexed by last MOZ crawling duplicate content errors
In the last crawler issues report from MOZ I can see many many pages listed as duplicate content with 0 duplicate urls. Like this: http://imgur.com/fbikRVq I am puzzled, what does it mean?
Moz Bar | | max.favilli0 -
Link to hotels on http://moz.com/mozcon doesn't work
Hi The link to the hotel for Mozcon 2015 doesn't work - seems like its the 2014 link still in place. Thanks Andy
Moz Bar | | Andy-Halliday0 -
Moz keyword search took
I don't understand how moz keyword search tool works and how to interpret serp analysis. Is there any tutorial on how to use it?
Moz Bar | | zsyed0