Unsolved Site Crawler not working but on-demand crawler working
-
Hi,
In Moz pro, when using Site crawler (or recrawl), we are seeing message site is banned. But when using on-demand crawler, it could generate report successfully.
I just like to know if in both these cases, it is roberbot that is used!
And kindly note, site crawler was perfectly working before. So the required setup is already in place from long time. Site crawler ban issue started appearing from nov/dec 2023. .
Could you please us understand how could we possibly make site-crawler work?
I am happy to provide more details if you need any.Thanks
-
Hi,
This question requires help from MozPro.
Site Crawler is not working because it is missing request header 'user-agent' when we investigated the logs in our system and it got banned because of this reason.
On-demand crawler is still working because it has request header 'user-agent' and our system approved it hence able to generate report.Could you please look into this issue of no-user-agent request header?
Your response is much appreciated.Thanks
-
Hi,
I will double-check with firewall settings in our servers. Could you please share moz-pro site-crawler roger bot IP addresses/range? We will verify against our firewall rules.
Thanks
Shashi -
I am looking for roger bot site crawler IP addresses Please provide.
Thanks
-
@Aditi_08
Could you please help me on how to get IP addresses of Site Crawler? Just please note, Site Crawler is working before November so IP addresses were not blocked.Like it is mentioned before,
- no change in robots.txt
- no issue with rate limiting
- no changes in site-crawler configuration
-
@gilesd If you're experiencing issues with Moz Pro's Site Crawler showing that the site is banned while the On-Demand Crawler works fine, it might be due to changes or updates since November/December 2023. Both tools likely use the same crawler, "rogerbot," but differ in their operational schedules. The problem could be due to rate limiting or blocking by your server, IP blocking, changes in your robots.txt file, or updates in the Site Crawler configuration. To resolve this, check your robots.txt file to ensure it allows Moz's crawler, review server logs and firewall settings to ensure the crawler’s IP addresses aren’t blocked, and adjust rate limiting settings if necessary. Also, double-check the settings in Moz Pro to make sure there are no configurations causing the issue. If the problem persists, contact Moz support with detailed information about the error messages and any recent changes to your site’s configuration. Regular monitoring of your site’s interactions with automated tools and coordinating with your hosting provider can help prevent such issues in the future.
-
I am not sure why my reply not appearing here. Just for confirmation, replying again,
I like to confirm you -
There is no modification in Robots.txt
No issues with rate limit
Moz Pro settings are not changedWe are looking for your help to identify the issue.
Thanks
-
Thanks for your trouble shooting tips.
I assure you there has been nothing changed in robots.txt file or any settings in MozPro.
And there is frequency limit, Site Crawler triggers only once in 2 weeks.Thanks
-
Hi, gilesd
In Moz Pro, when using the Site Crawler or Recrawl, we also received a message indicating the site was banned. However, the on-demand crawler could generate the report successfully.
To address your question:
Robots.txt Configuration: Both the Site Crawler and on-demand crawler should be using the same robots.txt file unless there's been a recent change. Ensure your robots.txt hasn't been updated to block specific user agents.
IP Blocking or Rate Limiting: Some web servers or security settings might block or limit access based on IP or request frequency. The Site Crawler might be hitting these limits, whereas the on-demand crawler, being less frequent, avoids these blocks.
Moz Pro Settings: Double-check the Moz Pro settings to see if there have been any changes or updates to how the Site Crawler operates compared to the on-demand crawler. Any recent updates might have altered how the Site Crawler interacts with your site.
Thanks,
Hamza Zubair -
Hi, gilesd
In Moz Pro, when using the Site Crawler or Recrawl, we also received a message indicating the site was banned. However, the on-demand crawler could generate the report successfully.
To address your question:
Robots.txt Configuration: Both the Site Crawler and on-demand crawler should be using the same robots.txt file unless there's been a recent change. Ensure your robots.txt hasn't been updated to block specific user agents.
IP Blocking or Rate Limiting: Some web servers or security settings might block or limit access based on IP or request frequency. The Site Crawler might be hitting these limits, whereas the on-demand crawler, being less frequent, avoids these blocks.
Moz Pro Settings: Double-check the Moz Pro settings to see if there have been any changes or updates to how the Site Crawler operates compared to the on-demand crawler. Any recent updates might have altered how the Site Crawler interacts with your site.
Thanks,
Hamza Zubair
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unable to site crawl
Hi there, our website was revamped last year and Moz is unable to crawl the site since then. Could you please check what is the issue? @siteaudits @Crawlinfo gleneagles.com.my
Technical SEO | | helensohdg380 -
New blog site spam score is 40+ without any backlink
I have purchased a new domain ( Studytobecome.com ) from GoDaddy, before 15 days, and i just writing daily 1 article on my site, without any SEO, or backlinks, but now when I see in Moz spam score of my site after 15 days it shows 40+ without any links. How to reduce it, and whats the problem is, I don't understand.Please help me.5Vc6zl8
Moz Pro | | bhavierureu1 -
How to set up Rel=canonical in Joomla based sites
I've built a few sites using joomla (please don't tell me I should be using wordpress!!) and wondered how I can add the rel-canonical to these pages. I'm assuming it would come as a plugin or module but can't seem to find anything that works right for me. Anyone any ideas? Thanks in advance, Gordon
Moz Pro | | Gordon_Hall0 -
Site Not Indexing & SEOMoz Reporting ZERO On-Page Report Crawls
Any help on this would be MUCH appreciated. One of my sites, aironeairsolutionsinc.com, has recently been rebuilt and the pages tweaked for some basic optimization. Based on my experience, those tweaks (geared toward keywords with relatively low competition locally) usually bump my local sites up into the top 20 or 30 at worst. 3 weeks later, it seems my site is still not indexing with Google. In addition, I AM NOTICING THAT THE ON PAGE REPORTS IN SEO MOZ ARE NOT REGISTERING THAT ANY PAGES ARE BEING CRAWLED. Again, any help from Moz staff would be awesome! :} Thanks, Ricky
Moz Pro | | RickyShockley0 -
To Many Links on site
I've had an issue with to many links on the site. My drop down menu, secondary footer and footer. The report told me that I had 253 links on each page. I then programmed my secondary footer to dynamic and ran a crawl and my links reduced accordingly to 201. Then turned the footer into dynamic and ran a crawl with my links increasing to 1500. This also happened between each phase but en went away. Oddly enough, my domain authority increased as well as other factors in the crawl report. This too many links thing is driving me crazy. Please provide some guidance.
Moz Pro | | CHADHARRIS0 -
Internal links not showing in Open Site Explorer
So I'm working on a law firm site and looking at the links for pages in OSE. For practice areas, the links to each practice area are in the left hand menu on every page of the site. Can anyone help me with this question: Example: http://www.comitzlaw.com/personal-injury/car-accidents.html When I plug this URL into OSE, it only shows one linking page, www.comitzlaw.com/practice-areas.html, yet there is a link to this on every other page in the site. When I plug in a random competitors page, www.lesagelblaw.com/Personal-Injury-Overview/Car-Accidents.shtml, it does show all the internal pages linking to it. Since I'm not using a flash menu or javascript, any ideas as to why no internal links are showing up in OSE? Even when I plug in the main URL for the home page, it only shows 4 other internal pages linking to it, yet there is a link on every page. What am I doing wrong?
Moz Pro | | c2g0