Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Local
Hey Does anyone know when Moz Local will be available in the UK? Please share your experience. I'd like to know how it's actually helped your business and client base. Thanks so much Gary
Moz Pro | | GaryVictory0 -
Setting a campaign in SEOmoz
Hi All, when i set a new campaign i am asked to decide if to use: domin-name.com Or www.domain-name.com Can someone please explain the different in terms of the campaign If i use: domain-name.com - will the campaign run on www.domain-name.com too? Thank you SEOwise
Moz Pro | | iivgi0 -
SEOMoz Private Message Settings
Hi all, Is there a way to be notified (to my personal email address) when I get sent a private message? Thanks,
Moz Pro | | Unity
Davinia0 -
This stuff works - but be patient
..or subtitled "This is how I did it" Managed to get clients second most important keyword to #1 on Google recently (most important is about 5 or 6, am getting there with that one) 🙂 Yes, we all know 'rankings aren't important, traffic/searches are' but you know what - when a Client sees their keyword at #1 it kinda helps getting paid. Thus, in summary, this is how I did it. I don't think there's anything earth shattering here but it might help a few out as I see so many 'where do i start' type posts. 1 - Do everything within Webmaster Tools that is humanly possible. Sitemaps, fix errors, preferred domain the lot. Spend ages here! 2 - Get your on page optimisation sorted out. Decide on THE main keyword for page, fire it into your campaign and test it. Do not give up until it's an A and you've done ALL the tips/hints (yes, even bold text on the page.) It's obvious but you have to tell a Search Engine exactly what the page is about - and give it a few hints too. 3 - Links - ahhh, good old links the bain of our lives. This site was struggling for links so i bought some - yes, the shame of it. I still think the best directories are a good starting point, spent about £400 or so. (BOTW, JoeAnt, HotVsNot etc,) Also, find low hanging fruit places for links. See where top 3 or 4 competitors are getting links from (via OSE) and get your site linked there too. These aren't always chargeable. 4 - Get rid of as many errors you can from Crawl Diagnostics (Roger) and do everything you can to ensure page(s) load as quick as possible. e.g. for images, resize them and reduce colour depth. 5 - Go over Steps 1-4 again and again AND AGAIN. I think that's about it (will add anything if i think of it), this all took about 4-5 months - not to do the SEO work but for Google (or any SE) to recognise it all, so thanks to all at SEOmoz, the blogs and this forum for all the assistance. (Just need to get Linkscape updated quicker now folks - i couldn't resist that one!)
Moz Pro | | Capote1 -
How well is your Internet marketing working for your business?
Not asking about specifics of money, but this is just a general question to see what kind of return people get for their Internet marketing time investment.. I would say I'm getting over 40,000 hits per year for one business which is holiday cottages but I only own one cottage and act as an agent for three other clients. Because the campaign is working well for me at the moment I get probably 95% occupancy throughout the year, which you would have no chance of accomplishing without SEO intensive input. If I had to pay £0.50 a click approximately obviously it wouldn't be worthwhile, especially when you realise that a percentage of the traffic and not related to your product. I also have a second business which is installing wireless security alarms over the Internet and once again approximate traffic at the moment is almost approaching 40,000 again per year. Now I'm sure there are some big hitters out but my general question is how productive, is you're organic results and Internet marketing for your personal business in comparison to what it would cost you if you had to pay per click? I'm kind of wondering how well the Internet is working people on this forum for their particular businesses? Good, Bad ? whats your story ? Cheers alan
Moz Pro | | whitbycottages0 -
MOZ Crawler only crawling one page per campaign
We set up some new campaigns, and now for the last two weekly crawls, the crawler is only accessing one page per campaign. Any ideas why this is happening? PS - two weeks back we did "upgrade" the account. Could this have been an issue?
Moz Pro | | AllaO0 -
About the rankings report in the Pro Dashboard, does it track the ranking of every page on a root domain, or just the home page or whichever page you set up the campaign with?
I noticed that one of the pages on my root domain has a #5 rank for a keyword, yet the ranking report says that there are no results in the top 50. So I am assuming it is only tracking the home page. That is one thing I liked about the Rank Tracker, that it would find any page that was ranking on a root domain. Thanks, Lara
Moz Pro | | larahill0 -
Campaign keywords (48) and on page report (17)
I go into my campaign keywords and have 48, but when I view my on page report only 17 show up? All keywords have been in there since last Friday except 2-26-11, 4 I added this Friday 3-4-11.
Moz Pro | | SmallFry340