Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Robots.txt blocking Moz
-
Moz are reporting the robots.txt file is blocking them from crawling one of our websites.
But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems.
We have never come up against this before, even with this site.
Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error.
Can anyone enlighten us to the problem please?
http://www.wychwoodflooring.com
-Christina
-
Hi Nigel
Neither, they use server side filtering.Regards- David
-
Hi David
That's great news!
As a matter of interest, where did they block it? as it's not in the Robots.txt - was in in htaccess.txt?
Regards
Nigel
-
Nigel,Thanks for the reply, the cgi-bin folder is never used by any of my sites but I put this in just as a matter of course, the folder would normally contain old cgi scripts so would not usually affect the crawling of a robot in any case.The reason for the problem turns out that our host had blocked rogerbot along with several other malicious bots, they have now lifted this block and the site is able to be crawled.- David
-
Hi Christina
I don't know how your site is set up but I can see that for some reason you are blocking access to the cgi-bin
If that directory contains files that execute php or other permissions then that may well be your problem. It's the only directory you are blocking and since I haven't seen other Robots.tx blocking it, then I would hazard a guess that this is the root of your problem.
Robots.txt
User-agent: * Disallow: /cgi-bin/ Sitemap: http://www.wychwoodflooring.com/sitemap.xml
Regards
Nigel
-
Our hosting provider has banned Rogerbot as they see it as problematic!!!!
They are a great hosting provider so this is going to be a difficult one.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Need Moz SEO Wordpress Plugin With API
Re: Moz WordPress Plugin? Hi guys,
Moz Pro | | mrezair
I need some Moz SEO Wordpress Plugins For my website working with Moz API. I've already found Moz DA-PA Checker plugin Moz DA-PA Checker But Need SEO Plugins too. Any Suggestion will be appreciated.0 -
Unsolved Does Moz Pro include Moz Local
My client has bought about six Moz Local accounts and are pleased with results. We have not yet used your Moz Pro program. The client might be interested in switching to the Moz Pro if those Moz Local accounts can be included into it. Please let me know as soon as possible. Thanks!
Moz Pro | | gallowaywebteam0 -
What's the best way to search keywords for Youtube using Moz Keyword explorer?
I want to optimize my youtube channel using identified keywords, but I'm concerned that the keywords I'm identifying work well for SERP's but might not be how people search in Youtube. How do a distinguish my keywords to be targeted for Youtube?
Moz Pro | | Dustless0 -
Block Moz (or any other robot) from crawling pages with specific URLs
Hello! Moz reports that my site has around 380 duplicate page content. Most of them come from dynamic generated URLs that have some specific parameters. I have sorted this out for Google in webmaster tools (the new Google Search Console) by blocking the pages with these parameters. However, Moz is still reporting the same amount of duplicate content pages and, to stop it, I know I must use robots.txt. The trick is that, I don't want to block every page, but just the pages with specific parameters. I want to do this because among these 380 pages there are some other pages with no parameters (or different parameters) that I need to take care of. Basically, I need to clean this list to be able to use the feature properly in the future. I have read through Moz forums and found a few topics related to this, but there is no clear answer on how to block only pages with specific URLs. Therefore, I have done my research and come up with these lines for robots.txt: User-agent: dotbot
Moz Pro | | Blacktie
Disallow: /*numberOfStars=0 User-agent: rogerbot
Disallow: /*numberOfStars=0 My questions: 1. Are the above lines correct and would block Moz (dotbot and rogerbot) from crawling only pages that have numberOfStars=0 parameter in their URLs, leaving other pages intact? 2. Do I need to have an empty line between the two groups? (I mean between "Disallow: /*numberOfStars=0" and "User-agent: rogerbot")? (or does it even matter?) I think this would help many people as there is no clear answer on how to block crawling only pages with specific URLs. Moreover, this should be valid for any robot out there. Thank you for your help!0 -
Why is MOZ and Google search Volume so different?
A search term in MOZ shows the monthly search volume to be 49K. In Google, the same term shows the search volume at only 1300 monthly searches. Which do I trust? Thanks, Don
Moz Pro | | rcman0 -
Moz Pro vs. Moz Analytics
Beta version has been out for awhile. Curious to know what people think of Moz Analytics. Personally, I'm having a hard time making the switch from Pro to Moz Analytics because I'm so used to the navigation and layout of Pro. What does everyone else think of the new Moz Analytics? Advantages? Disadvantages?
Moz Pro | | EEE31 -
White Label Version of Moz Pro?
I may have missed it, but is there a white label option for Moz Pro? I have a few clients that would like to login and have a look at competition, rankings, etc. Perhaps something like WebCEO or MySEOtool?
Moz Pro | | TysonDowns0