Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does increase in DA increase Moz Trust score?
I have checked many website which in my office. Have a confusion Does increase in DA increase Moz Trust score? I have a website https://emocrest.com/ which is new one and Moz Trust score as of now is 1 only. Well, most of the projects I'm handling in our office having 2,3,4 in the range. How this MT value is calculated and does it got any relation with the change in Domain Authoiry or the page authority of a website?
Moz Pro | | nazfazy1 -
Why is my MOZ ranking better but my ranking worse
last week, I made all the recommended changes for my client's website, and our grade according to MOZ got better, but our rankings dropped?
Moz Pro | | lfrazer0 -
Please set somethings straight
as a recent majesticSEO subscriber, I feel like i have either been mislead or something isn't right. For some reason I was thinking it was best to have a single link from a single domain, and that by have too many links from a single domain (backlinks) the ratio between referring domains and backlinks was a factor in pagerank. In domain analysis here, it would seem that is not that case as it puts a nice big green check mark by the person with most links, whether they are from 1 domain or multiple domains. Is this a good thing? Here's an easy example. I am a website designer, and I have the ability to link to my website in the footer of every website I design either by image or by text. If I do that, obviously I would have a link from every page on the site, but only from one root domain. Is this a good thing or a bad thing. My work around is to have a link in the footer that looks like this "website design by homecutcreative.com" that links to an internal page that has one link to my site, therefore keeping my referring domain : backlink ratio to a 1 which I was under the assumption was the best possible situation. Thanks for your help.
Moz Pro | | jonnyholt0 -
Are labels useful in managing keywords in SEOMoz Campaigns?
I am using SEOMoz for only one domain. Should I track keywords under only one campaign, or should I utilize the different campaigns for different types of keywords under the same domain? Is it easier to keep all keywords for a domain in one campaign? Is it helpful to assign labels to keywords within the campaign? What types of labels should I create?
Moz Pro | | djlittman1 -
Moz bar down?
Is the Mozbar not working for link data? Whats up with moz I signed up for a free trial and nothing works....
Moz Pro | | JohnWalker0 -
How to setup campaign
Is it better to specify www.site.com or *site.com for SEO Moz campaign? assuming that I do not have any subdomains except www, in the Web Master tools I have selected preferred address as www.site.com and I have redirect in my .htaccess from site.com to www.site.com The reason I am asking is because I see different SeoMoz trust and other stats for root domain and www subdomain. I would think they should be the same, given the fact that I only use www?
Moz Pro | | SirMax0