Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Mobile Rankings for me and competitors disappear in Moz but still ranking.
Hi, I'm trying to create an SEO report for my client however their mobile rankings have completely disappeared for Google Mobile rankings. Competitors rankings have disappeared as well. Could this be a Moz configuration issue?
Moz Pro | | baddjuju0 -
Robots.txt blocking Moz
Moz are reporting the robots.txt file is blocking them from crawling one of our websites. But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems. We have never come up against this before, even with this site. Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error. Can anyone enlighten us to the problem please? http://www.wychwoodflooring.com -Christina
Moz Pro | | ChristinaRadisic0 -
Moz analytics telling me I have duplicate content issues - how to fix this?
Hey guys, Okay I ran into moz analytics - I have I have 199 Issues, priority issues are showing 38 Duplicate page content. I began looking into the URL's and from what I have noticed from all the urls are showing me a common theme. The urls are pointing to my blog pages - my blog is using wordpress. What iv noticed is the urls all have "Tag" in it Here are 3 examples that I have found. All url's take me to a blank page: Does anyone know what the solution is to fixing this? I read the article for duplicate content covering 301 redirects and Rel=Canonical tags - I'm wondering if this would need to be considered in this case? However I find it confusing that these pages for to a blank page. https://www.zenory.com.au/blog/tag/dysfunctional-relationships/ https://www.zenory.com.au/blog/tag/change/ https://www.zenory.com.au/blog/tag/intuitive/ Appreciate some assistance.
Moz Pro | | edward-may0 -
Why is my MOZ ranking better but my ranking worse
last week, I made all the recommended changes for my client's website, and our grade according to MOZ got better, but our rankings dropped?
Moz Pro | | lfrazer0 -
What Does the SEO Moz Crawler Take Into account?
I'm working on a page that has links from some decent pages pointing to it, but a lot of them are low-value blog comments. So I'm pretty sure that its Page Authority is higher than it should be, compared to where it's ranking. Does SEO moz take the type of link into account? i.e. if it's a footer link, blog comment, or forum signature link; this should carry less weight than a link in the content of the page itself, as it does with Google.
Moz Pro | | seanmccauley0 -
Creating a campaign for a folder is not possible
Hi There! I just signed up for a free pro account to see whether it is something for us as a company 🙂 I tried to set up two campaigns. With the first for our root domain (koopgoedkoop.nl) it all went perfect. I am now trying to set up a campaign for our BBQ page (http://www.koopgoedkoop.nl/tuin/barbecues.html) but I can't? Is it because our URL ends with .html? Here's a screenshot: http://bit.ly/x2zdCd Thanks for your help in advance! Regards, Do
Moz Pro | | vidaXL0 -
How do I get the SERP overlay tool to work?
I have the SEOMoz toolbar installed. In the settings I have a tick next to Display SERP Overlay. When I first activated this it showed up but with no data just continually searching for a long time. Now it is not showing up at all. This is such a great tool, how can I get it working? Thanks, Daniel.
Moz Pro | | iSenseWebSolutions0