Moz campaign works around my robots.txt settings
-
My robots.txt file looks like this:
User-agent: *
Disallow: /*?
Disallow: /search
So, it should block (deindex) all dynamic URLs.
If I check this url in Google:
site:http://www.webdesign.org/search/page-1.html?author=47
Google tells me:
A description for this result is not available because of this site's robots.txt – learn more.
So far so good.
Now, I ran a Moz SEO campaign and I got a bunch of duplicate page content errors.
One of the links is this one:
http://www.webdesign.org/search/page-1.html?author=47
(the same I tested in Google and it told me that the page is blocked by robots.txt which I want)
So, it makes me think that Moz campaigns check files regardless of what robots.txt say? It’s my understanding User-agent: * should forbid Rogerbot from crawling as well. Am I missing something?
-
That worked, thanks!
-
Thanks Abe.
I guess I'll try this:
Useragent: Rogerbot
Disallow: /*?
Because if I use Disallow: / I'll lose my current Moz reports because Rogerbot will just ignore all my file, right?
-
Hello Vince, thank you for reaching out to us! This seems quite odd, our crawler usually obeys all robots.txt files. Let's try this. Add this code to your robots.txt:
Useragent: Rogerbot
Disallow: /
This should specifically instruct us to follow these rules. Once you have tried this, if it does not work, please send an email to help@moz.com and we will have our engineers dig in a bit further. Sorry for the inconvenience, I hope the above fix works for you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Solve a Lost GA Connection - can't get it to work
Hi, I followed all steps on https://moz.com/help/moz-pro/getting-started/troubleshoot-google-analytics?_ga=2.15852085.2007735045.1676157945-122376721.1676157945 below Solve a Lost GA Connection multiple times but I keep getting the message "Our connection to your Google Analytics account was lost. Don't worry, you won't lose any data. Please follow the steps in our guide to reconnect.
Moz Pro | | silvansoeters
" I am on the free trial and would obviously like to connect to GA to see how everything is working. Any help would be greatly appreciated. Thanks.0 -
New to Moz and wanted a bit of help with my report
Hi, I have used the MOZ report to analyse one of my friends sites and I wanted to query a few warnings it highlighted and I just wanted people's thought on how important they thought these were: The first is dupliate descriptions/titles. This is mainly down the e-commerce pages. Fist duplicate content:
Moz Pro | | dannylancs
On some pages the description is identical and all that is different is the title and picture, is this an issue? Duplicate pages:
Due to the way the website folder structure/catergories has been created some pages are identical but because the product comes under 2 cetergories there is 2 seperate pages, should we use the canonical on one of the pages? Also regarding the canonical tag, they have put link rel="canonical" on every page and got it to point at itself, so not really being used in the way it is meant to be. Could something like this cause any harm? The final thing is internal linking back to the homepage. If for example the homepage is http://www.test.com, when linking back is it best to put the full URL over "index.html" even though they are the same page? Any help really appreciated Dan0 -
Duplicate Titles Shown in Moz Analytics
Hello, My Moz analytics show over 50 duplicate titles. This because my website title is shown on each page (eg About us | Social Engagement, Contact us | Social Engagement, etc.) Is this really an error to be fixed, that otherwise will impact rankings? Or can I leave it as it is? I don't mind having my website title on each page, but if its worth removing it due to duplicate issues impacting ranking, I will remove 🙂 In this case, does someone have any suggestion on how to do this? Which files should I modify? (Im using Wordpress CMS, and I dont see each page code) Thanks in advance.
Moz Pro | | socialengaged
Regards, Eugenio | Social Engagement (http://socialengagement.it)0 -
Moz reporting incorrect ranking
Moz is reporting no ranking on one of my keywords, but when I search that keyword on Google we are on the first page. What would cause Moz to report that incorrectly?
Moz Pro | | SpearOne0 -
How to setup campaign
Is it better to specify www.site.com or *site.com for SEO Moz campaign? assuming that I do not have any subdomains except www, in the Web Master tools I have selected preferred address as www.site.com and I have redirect in my .htaccess from site.com to www.site.com The reason I am asking is because I see different SeoMoz trust and other stats for root domain and www subdomain. I would think they should be the same, given the fact that I only use www?
Moz Pro | | SirMax0 -
Seomoz bar: No Follow and Robots.txt
Should the Mozbar pickup 'nofollow" links that are handled in robots.txt ? the robots.tx blocks categories, but is still show as a followed (green) link when using the mozbar. Thanks! Holly ETA: I'm assuming that- disallow: myblog.com/category/ - is comparable to the nofollow tag on catagory?
Moz Pro | | squareplug0 -
Campaign Manager Keyword Rankings all messed up???
My campaign in SEO Moz Campaign Manager has just ran it's weekly crawl, and all my keyword rankings are messed up. 18 keywords have apparently dropped by massive amounts, but when I do a physical search, they haven't moved. Also Keyword Tracker agrees with the physical search, so why is the Campaign Manager showing the wrong rankings all of a sudden??
Moz Pro | | gdavies090319770 -
No new campaigns after upgrading?
Hey all, anyone know how to add new campaigns in the pro dashboard after you upgrade? I went from pro to plus pro to get 12 campaigns, but it won't let me add anymore. It says I am a plus pro member and all, but there's no option to add more campaigns, anyone know what to do here?
Moz Pro | | DanDeceuster0