Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Robots.txt blocking Moz
-
Moz are reporting the robots.txt file is blocking them from crawling one of our websites.
But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems.
We have never come up against this before, even with this site.
Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error.
Can anyone enlighten us to the problem please?
http://www.wychwoodflooring.com
-Christina
-
Hi Nigel
Neither, they use server side filtering.Regards- David
-
Hi David
That's great news!
As a matter of interest, where did they block it? as it's not in the Robots.txt - was in in htaccess.txt?
Regards
Nigel
-
Nigel,Thanks for the reply, the cgi-bin folder is never used by any of my sites but I put this in just as a matter of course, the folder would normally contain old cgi scripts so would not usually affect the crawling of a robot in any case.The reason for the problem turns out that our host had blocked rogerbot along with several other malicious bots, they have now lifted this block and the site is able to be crawled.- David
-
Hi Christina
I don't know how your site is set up but I can see that for some reason you are blocking access to the cgi-bin
If that directory contains files that execute php or other permissions then that may well be your problem. It's the only directory you are blocking and since I haven't seen other Robots.tx blocking it, then I would hazard a guess that this is the root of your problem.
Robots.txt
User-agent: * Disallow: /cgi-bin/ Sitemap: http://www.wychwoodflooring.com/sitemap.xml
Regards
Nigel
-
Our hosting provider has banned Rogerbot as they see it as problematic!!!!
They are a great hosting provider so this is going to be a difficult one.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Mobile Rankings for me and competitors disappear in Moz but still ranking.
Hi, I'm trying to create an SEO report for my client however their mobile rankings have completely disappeared for Google Mobile rankings. Competitors rankings have disappeared as well. Could this be a Moz configuration issue?
Moz Pro | | baddjuju0 -
Unsolved Rogerbot blocked by cloudflare and not display full user agent string.
Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
Moz Pro | | BB_NPG
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
Ben Cloudflare activity log, showing differences in user agent strings
2022-07-01_13-05-59.png0 -
How long does Moz takes to update DA and PA value?
I used to work daily on my office website https://quintdaily.com/ and follow link building strategy. however, since more than 40-50 days, there is no movement in my DA, as the PA has just lifted up. However, there is no changes in Domain Authority so far. Besides, other than this website, im working on location wise SEO projects, but, i can find those websites Domain Authority changed in less 20 days. Is there is time period in changing the Domain Authority once it touched DA 30+?
Moz Pro | | seochris172 -
How long does it takes for Moz pro to find backlinks and increase my score?
Hi, I'm running a website for tourism & travel called Visit Guide or دليل السفر and i have many backlinks from many websites ( DoFollow & NoFollow ) They are older than 10 days and Moz has not discovered them yet nor found them? How to push Moz Pro to find my backlinks? or how to SpeedIndex my backlinks in other words?
Moz Pro | | VisitDotGuide0 -
What's the best way to search keywords for Youtube using Moz Keyword explorer?
I want to optimize my youtube channel using identified keywords, but I'm concerned that the keywords I'm identifying work well for SERP's but might not be how people search in Youtube. How do a distinguish my keywords to be targeted for Youtube?
Moz Pro | | Dustless0 -
Why is MOZ and Google search Volume so different?
A search term in MOZ shows the monthly search volume to be 49K. In Google, the same term shows the search volume at only 1300 monthly searches. Which do I trust? Thanks, Don
Moz Pro | | rcman0 -
How you can manipulate your MOZ DA
I have become frustrated at MOZ in the last few months, none of my backlinks have made it into the index. Old back links. Long story short, I figured out the issue and I figured out how anyone can manipulate their DA. I wrote a blog post about it here, http://blog.dh42.com/manipulate-moz/
Moz Pro | | LesleyPaone1 -
Moz & Xenu Link Sleuth unable to crawl a website (403 error)
It could be that I am missing something really obvious however we are getting the following error when we try to use the Moz tool on a client website. (I have read through a few posts on 403 errors but none that appear to be the same problem as this) Moz Result Title 403 : Error Meta Description 403 Forbidden Meta Robots_Not present/empty_ Meta Refresh_Not present/empty_ Xenu Link Sleuth Result Broken links, ordered by link: error code: 403 (forbidden request), linked from page(s): Thanks in advance!
Moz Pro | | ZaddleMarketing0