Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem to log into moz
Every time the moz logs me out from the account and then I can not log in. It shows on the left side my name like I am logged in and then when I want go to community suddenly I am not logged in. It offen shows 502 error. It was first doing on firefox, then I manage to log in chrome and now I had to log in private browsing.
Moz Pro | | Rebeca11 -
Why is my MOZ report only crawling 1 page?
just got this weeks MOZ report and it states that it have only crawled: Pages Crawled: 1 | Limit: 10,000 it was over 1000 a couple of weeks ago, we have moved servers recently but is there anything i have done wrong here? indigocarhire.co.uk thanks
Moz Pro | | RGOnline0 -
Do the SEOmoz Campaign Reports follow Robots.txt?
Hello, Do the SEOmoz Campaign Reports (that track errors and warnings for a website) follow rules I write in the robots.txt file? I've done all that I can to fix the legitimate errors with my website, as reported by the fabulous SEOmoz tools. I want to clean up my pages indexed with the search engines so I've written a few rules to exclude content from Wordpress tag URLs for instance. Will my campaign report errors and warnings also drop as a result of this?
Moz Pro | | Flexcin0 -
Confused on setting up my domain in a campaign
I have a website that is www.example.com. When I first setup my campaign in SEOMoz, I missed the instructions that stated I need to set up my domain as it appeared in the browser. Thus, I set it up as example.com. When I realized my mistake, I setup a second campaign as www.example.com and am running them side by side to see the differences. My question has to do with the best setup. I have a blog that is a subdomain as blog.example.com. It appears that the www.example.com campaign is not crawling the blog pages but example.com is. Are there any downsides in setting up this campaign as example.com? I have a number of other sites with the same issue so would like to get this resolved before I setup the additional sites. Thanks for any assistance or insights into ways I may be "shooting myself in the foot" with the wrong domain structure in my campaign settings.
Moz Pro | | rfwood0 -
Ideal Campaign Set Up For Multiple Landing Pages Post Panda
I have a local floor cleaning business in the UK. I have various discrete services e.g. carpet cleaning, stone restoration and polishing (a page for each type of stone) that are delivered in distance geographical service areas. Each page is set up to target a "town/service" keyword. In each geographical location I have different competitors Should I set individual campaigns for each landing page or is one site wide campaign the way to go. Many thanks David Allen
Moz Pro | | davidallen3650 -
Campaign Reports Question
Hello everyone. I'm new to SEOMOZ and sort of a beginner with this SEO stuff but this does seem like the place to be to learn things. I spent quite a bit of time reading the blogs and found them very helpful - including everyone's comments. I'm not sure what to make of a couple of my campaign reports. I'm going through my campaign reports and have a few questions. I guess the easy way is to break this down. Question 1: Crawl Errors: I"m getting duplicate content/page errors for the following pages. http://www.njsigncrafters.com and http://www.njsigncrafters.com/index.php Am I losing a bit of google points because of this and do I fix it by using a redirect? Question 2: When I look at my on-page analysis I'm getting grades of A-F. I'm figuring maybe I should have some recommendations about what to do but SEOMOZ gives me a grade and then leaves it at that. Besides reviewing my on-page factors am I missing something simple that I"m supposed to click on to see recommendations? I suppose once I get used to the way things work I'll get this SEOMOZ site figured out.
Moz Pro | | dpdeleon10 -
Drop in number of Pages crawled by Moz crawler
What would cause a sudden drop in the number of pages crawled/accessed by the Moz crawler? The site has about 600 pages of content. We have multiple campaigns set up in our Pro account to track different keyword campaigns- but all for the same domain. Some show 600+ pages accessed, while others only access 7 pages for the same domain. What could be causing these issues?
Moz Pro | | AllaO0 -
SEOMOZ Stats dont work out
Hi, When I check my mozstats for the homepage it says the PA is 50 but the DA is 30, how can that be? I would expect them to either be the same or at least the DA to be higher then the PA. Cheers
Moz Pro | | activitysuper0