How to block Rogerbot From Crawling UTM URLs
-
I am trying to block roger from crawling some UTM urls we have created, but having no luck. My robots.txt file looks like:
User-agent: rogerbot Disallow: /?utm_source* This does not seem to be working. Any ideas?
-
Shoot! There may be something else going on. Give us a shout at help@moz.com and we'll see if we can figure it out!
-
FYI - I tried this and it did not work. Rogerbot is still picking up URL's we don't need. It's making my crawl report a mess!
-
The only difference there is the * wildchar. The string with that character will limit the crawler from accessing any URL with that string of characters in it.
-
What is the difference between Disallow: /*?utm_ and Disallow: /?utm_ ?
-
Hi there! Tawny from the Customer Support team here!
You should be able to add a disallow directive for that parameter and any others to block our crawler from accessing them. It would look something like this:
User-agent: Rogerbot
Disallow: ?utmetc., until you have blocked all of the parameters that may be causing these duplicate content errors. It looks like the _source* might be what's giving our tools some trouble. It looks like Logan Ray has made an excellent suggestion - give that formatting a try and see if it helps!
You can also use the wild card user-agent * in order to block all crawlers from those pages, if you prefer. Here is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt We always recommend checking your robots.txt file with a handy Robots Checker Tool once you make changes to avoid any nasty surprises.
-
Skyler,
You're close, give this a shot:
Disallow: /*?utm_
This will be inclusive of all UTM tags regardless of what comes before the tag or what element you have first.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Performance Metrics crawl error
I am getting an error:
Product Support | | bhsiao 0
Crawl Error for mobile & desktop page crawl - The page returned a 4xx; Lighthouse could not analyze this page.
I have Lighthouse whitelisted, is there any other site I need to whitelist? Anything else I need to do in Cloudflare or Datadome to allow this tool to work?1 -
Moz cant crawl site?
We're getting an error saying Moz is getting an errors crawling our client's site, but when I've put this though Google Search Console I'm not seeing any issues - any suggestions?
Product Support | | Ramarketingrob0 -
My site crawl has been in progress since last week
Hi there, I've been waiting on my site crawl to complete since Friday (it's Tuesday now), but it still has the 'in progress' notification at the top. Is it normal for it to take over 3 days? Or is there something holding it up?
Product Support | | VAPartners0 -
Why is comment moderation blocking all my comments? There are no links in them.
For the past few days, every time I try posting a comment, no matter how long or short, how informative or simple, I've been getting the error message that the comment isn't being allowed. I only tried a few times, once on several different posts (not spamming, God forbid!), and I never include links in my comments... So what's wrong?! So frustrated, I am so not up to this right now. Why can't I share my insights and questions on Moz anymore? Gaaaaaaa!
Product Support | | whiteonlySEO1 -
Both campaigns are now useless due to URL rewrite?
I have two campaigns on Moz and they were doing fine until I made the decision to rewrite my URL to remove www so, www.thing.com becomes thing.com Moz sees this as a error it seems and I am now getting error code 902. I tried to change my campaign setting but it won't let me change the URL because it's got historical information that doesn't pertain I guess. What should I do? Was it a mistake to remove the www? Thanks for any advise, Greg
Product Support | | Banknotes0 -
200 mozpoints : Removal of "nofollow" from first custom URL on profile but first link is nofollow...
Hello, I try to help Moz community and hope to get 200 mozpoints a day 🙂 When I analyze a profile from a member with more than 200 mozpoints, I see two links to custom URL link : one is added nofollow on a blank image, and one follow with anchor text. But it appears that the tests show that in this case the second link is not taken into account... Why not remove the first link so ? Here is the code: xxx: http://www.example.com/
Product Support | | Bigb060 -
Crawl errors are still shown after fixed
Fixed long ago "title too long" and some 404 errors, but still keep on showing on error statistics
Product Support | | sws10 -
Cannot create campaign because Moz doesn't recognize my URL
I have a new url, and I'm trying to create a new campaign for it. But in first step when i enter the domain, an error message pops up saying the url is invalid. could you help?
Product Support | | ALLee0