How to turn off automated site crawls
-
Hi there,
Is there a way to turn off the automated site crawl feature for an individual campaign?
Thanks
-
No problem!
Thanks for the suggestion!
I will pass that on as feedback
Eli
-
Hi Eli,
Thanks for your response.
I'd really suggest that the functionality to prevent the site crawl feature is built into the campaign tool. It would take dev effort to disalow the Moz bot where I'm based.
Thanks
-
Hey!
Thanks for reaching out to us!
Moz uses a crawler called Rogerbot to crawl your site and populate the "Site Crawl" section of your Campaign. Roger isn't a search engine index, so if URLs aren't blocked by the robots.txt file, then Roger will crawl them!
If you'd like to remove pages from the crawl, you can disallow them in the robots.txt file. Here's what that exclusion would look like if you were wanting to keep him from crawling any pages on your site:
User-agent: rogerbot
Disallow: *I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links between my two sites
I have a branded site (corporate site) that has links to my blog site. I am trying to locate all the links between the two. Most importantly from the BRANDED site to my BLOG. I've tried the inbound link but it shows I have none. What or how am I searching wrong?
Moz Bar | | Bwaples0 -
If we put the disavow links in google, does MOZ crawl the same links?
I have put bad or spam links in disavow file, but still showing in MOZ backlinks. So, I want to know that Why is MOZ not removing the spam links from their system?
Moz Bar | | insidewebanalytics0 -
Why is on-demand crawl reporting numbers that are very different than the regualr site crawl by Moz?
When I look at the site crawl available through my dashboard, it reports 22.9K pages crawled. I just ran a on-demand crawl and it is only reporting 6K pages crawled. Why is the number so much lower?
Moz Bar | | ctripp10100 -
Is there a way to export all your crawl errors for multiple Moz campaigns at once?
We're looking for a simple way to export all crawl errors for our Moz campaigns. More than likely we could use the API, but was wondering if there was any functionality already built into Moz for exporting all crawl errors.
Moz Bar | | ReunionMarketing0 -
Moz Crawl Report showing non-existent Duplicate Errors since new reporting layout
Hi Moz Community, Since Moz changed to the new style of Crawl report, we've seen a jump in duplicate errors for our site. These duplicate errors do not exist and were not present on the Crawl reports before the report change and also we have not made any changes to the flagged pages on our site since then either. When you download the report data in csv it appears that the Moz report is mixing up data for two or more pages on the site. e.g.in csv for 'Page1' data, it will show the meta description for 'Page2' and 'Page2' shows that for 'Page1', so this then gets flagged as duplicate, however looking at the actual Meta description assigned onsite, both Page 1 and Page 2 are completely unique. Has anyone else experienced this and Moz Team - are you looking into this? Thanks, V
Moz Bar | | WWTeam1 -
Not able to see results of HTTPS site on OSE
Hi mozzers, We have switched our website from http to https. But OpenSiteExplorer seems to support only http websites. Tell me if I am wrong.
Moz Bar | | Fancydad0 -
Moz crawl sees meta description but there are none
I have a new site I have run a starter crawl on. The crawl came back saying some of the pages do indeed have meta descriptions. When I go to the same page and use the Moz Chrome toolbar it says they do not have meta descriptions. I also know they do not have meta descriptions. Are there any instances in which the Moz crawler would see them when they are not there?
Moz Bar | | SBXMedia0 -
How do I get Moz to crawl my site?
If I'm not mistaken, Moz will crawl my site once a week and let me know of any errors, notices etc.. But can I ask them to crawl once I have updated all the errors? or do I have to wait for them to do this automatically?
Moz Bar | | david.smith.segarra0