How to turn off automated site crawls
-
Hi there,
Is there a way to turn off the automated site crawl feature for an individual campaign?
Thanks
-
No problem!
Thanks for the suggestion!
I will pass that on as feedback
Eli
-
Hi Eli,
Thanks for your response.
I'd really suggest that the functionality to prevent the site crawl feature is built into the campaign tool. It would take dev effort to disalow the Moz bot where I'm based.
Thanks
-
Hey!
Thanks for reaching out to us!
Moz uses a crawler called Rogerbot to crawl your site and populate the "Site Crawl" section of your Campaign. Roger isn't a search engine index, so if URLs aren't blocked by the robots.txt file, then Roger will crawl them!
If you'd like to remove pages from the crawl, you can disallow them in the robots.txt file. Here's what that exclusion would look like if you were wanting to keep him from crawling any pages on your site:
User-agent: rogerbot
Disallow: *I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz is only crawling 2 pages
Hi, I found a similar thread, but it did not provide a clear-cut answer. We have had this campaign running for over a year, and we are always adding content to the website, but Moz is only ever able to crawl 2 pages, Screaming Frog only picks up 12, but I know there is a lot more than that. None of our pages are set to no-index, so I do not know what is causing this. Welcoming any ideas/solutions. Thanks
Moz Bar | | GavinAdv0 -
Crawl Test : Error attempting to request HTTPS page
Hallo When I launch the crawl report I get csv file with this error : 804 : HTTPS (SSL) error encountered when requesting page.
Moz Bar | | micvitale
Error attempting to request page; see title for details. Website is https://bastabollette.it0 -
Weird 404 in Crawl Diagnostics
I'am getting a lot of 404 errors (196 to be precise ) - but their pattern is weird.
Moz Bar | | oorbo
The page that the crawler is trying to find is (e.g):
http://www.oorbo.com/item/asufa-israeli-design-shop**/www.oorbo.com.
the linking page is** http://www.oorbo.com/item/asufa-israeli-design-shop meaning it adds to the end of the link the root URL - /www.oorbo.com. This happens in all 196 cases - trying to find a page http://www.oorbo.com/some-page/www.oorbo.com from a refferer page http://www.oorbo.com/some-page. Obviously this pages do not exist, and it's getting a 404. I've look into the pages themselves and digged into their code - It doesn't seem that the bad link is any where on the page. Did anyone came across this kind of issue? any one can point me to a solution ?0 -
Moz crawl issues: All pages keep resolving to our "cookies not enabled" page
Upon running the Moz Pro site crawler, I noticed that I received quite a bit of duplicate titles along with 302 redirects (which is our site creating a temporary 302 to our "cookies not enabled" page). How would I get around the crawler being redirected to this page? I've never ran across this issue before, despite using the crawler with sites that use the same framework as the one thats affected. Any ideas?
Moz Bar | | responsivelabs0 -
Correcting a 4xx on my crawl report
How can I correct a 4xx error on my crawl report. This page no longer exists. What can I do?
Moz Bar | | henne0 -
Not able to see results of HTTPS site on OSE
Hi mozzers, We have switched our website from http to https. But OpenSiteExplorer seems to support only http websites. Tell me if I am wrong.
Moz Bar | | Fancydad0 -
Did Google Turn Off the Rank Tracking API ??
I might be responding too soon on this... but has anyone been able to use the Moz rank tracker or any other web-based rank-tracking tool that uses the Google API since about noon today (NYC TIME)??
Moz Bar | | EGOL0 -
How do I get Moz to crawl my site?
If I'm not mistaken, Moz will crawl my site once a week and let me know of any errors, notices etc.. But can I ask them to crawl once I have updated all the errors? or do I have to wait for them to do this automatically?
Moz Bar | | david.smith.segarra0