Regular Expressions for Filtering BOT Traffic?
-
I've set up a filter to remove bot traffic from Analytics. I relied on regular expressions posted in an article that eliminates what appears to be most of them.
However, there are other bots I would like to filter but I'm having a hard time determining the regular expressions for them.
How do I determine what the regular expression is for additional bots so I can apply them to the filter?
I read an Analytics "how to" but its over my head and I'm hoping for some "dumbed down" guidance.
-
No problem, feel free to reach out if you have any other RegEx related questions.
Regards,
Chris
-
I will definitely do that for Rackspace bots, Chris.
Thank you for taking the time to walk me through this and tweak my filter.
I'll give the site you posted a visit.
-
If you copy and paste my RegEx, it will filter out the rackspace bots. If you want to learn more about Regular Expressions, here is a site that explains them very well, though it may not be quite kindergarten speak.
-
Crap.
Well, I guess the vernacular is what I need to know.
Knowing what to put where is the trick isn't it? Is there a dummies guide somewhere that spells this out in kindergarten speak?
I could really see myself botching this filtering business.
-
Not unless there's a . after the word servers in the name. The . is escaping the . at the end of stumbleupon inc.
-
Does it need the . before the )
-
Ok, try this:
^(microsoft corp|inktomi corporation|yahoo! inc.|google inc.|stumbleupon inc.|rackspace cloud servers)$|gomez
Just added rackspace as another match, it should work if the name is exactly right.
Hope this helps,
Chris
-
Agreed! That's why I suggest using it in combination with the variables you mentioned above.
-
rackspace cloud servers
Maybe my problem is I'm not looking in the right place.
I'm in audience>technology>network and the column shows "service provider."
-
How is it titled in the ISP report exactly?
-
For example,
Since I implemented the filter four days ago, rackspace cloud servers have visited my site 848 times, , visited 1 page each time, spent 0 seconds on the page and bounced 100% of the time.
What is the reg expression for rackspace?
-
Time on page can be a tricky one because sometimes actual visits can record 00:00:00 due to the way it is measured. I'd recommend using other factors like the ones I mentioned above.
-
"...a combination of operating system, location, and some other factors can do the trick."
Yep, combined with those, look for "Avg. Time on Page = 00:00:00"
-
Ok, can you provide some information on the bots that are getting through this that you want to sort out? If they are able to be filtered through the ISP organization as the ones in your current RegEx, you can simply add them to the list: (microsoft corp| ... ... |stumbleupon inc.|ispnamefromyourbots|ispname2|etc.)$|gomez
Otherwise, you might need to get creative and find another way to isolate them (a combination of operating system, location, and some other factors can do the trick). When adding to the list, make sure to escape special characters like . or / by using a \ before them, or else your RegEx will fail.
-
Sure. Here's the post for filtering the bots.
Here's the reg x posted: ^(microsoft corp|inktomi corporation|yahoo! inc.|google inc.|stumbleupon inc.)$|gomez
-
If you give me an idea of how you are isolating the bots I might be able to help come up with a RegEx for you. What is the RegEx you have in place to sort out the other bots?
Regards,
Chris
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Huge Search Traffic Drop After Switching to HTTPS - No Recovery After Couple of Months
Hi In November, we have switched our website (https://www.insidermonkey.com) from HTTP to HTTPS. Initially, we noticed slight search traffic loss but later discovered it might be due to HTTPS switch. A month later we added the https version at search console, and then saw an immediate huge drop (about 25-30%). We discovered the problem might be due to poor redirection and noticed our redirects were 302s instead of 301s. To fix the problem, we implemented the 301 redirects and submitted the sitemap containing links to the old site at the new search console property (https). We've gone through points listed on the page below: https://support.google.com/webmasters/answer/6073543 We fixed the redirects to 301 Double-checked the sitemaps Made sure we had a properly installed SSL certificate (Now, we get A+ from https://www.ssllabs.com/ssltest/analyze.html?d=www.insidermonkey.com) Made sure we have no mixed-content errors (we don't have any issues at search console.) We only avoided implementing HSTS, in case we might want to switch back to HTTP.
Intermediate & Advanced SEO | | etakgoz
We had a small improvement in the following month, but our traffic did not fully recover. We wanted to test for the possibility to switch back HTTP by switching only 2 articles in our CMS to HTTP. Our traffic got worse, not only for those but for the whole site. Then we switched back those 2 articles to HTTPS again and implemented HSTS. It seems our search traffic getting worse day by day with no sign of improving. In the link below you can find the screenshot of our weekly search traffic between 1 October - 1 March. We are down from 500K weekly visitors to mere 167K last week. https://drive.google.com/open?id=1Y1TQbj_YtGG4NhLORbEWbvITUkGKUa0G Any ideas or suggestions? We are willing to get professional help as well. What is the way to find a proper consultant for such problem with relevant experience?0 -
Is it OK to dynamically serve different content to paid and non-paid traffic from the same URL?
Hi Moz! We're trying to serve different content to paid and non-paid visitors from the same URL. Is this black hat? Here's the reason we want to do this -- we're testing a theory that paid ads boost organic rankings. This is something we saw happen to a client and we want to test this further. But we have to have a different UX that's more sparse and converts better for paid. Thanks for reading!
Intermediate & Advanced SEO | | Horizon_SEO0 -
Site experiencing drop in Google rankings and organic traffic after redesign.
Hello, The company that I work for recently implemented a complete redesign for our company website. The former site was old, cumbersome and in desperate need of an update. We streamlined the site structure and made sure to redirect as many pages as we could find to new thematically related pages with 301 redirects. After the launch of our new site we saw a large upswing in "soft" 404 errors despite the fact that most of these pages do redirect upon inspection. So in relation to the soft 404s, for example, is it merely a matter of labeling them as fixed if they redirect properly, or could their be an underling issue with the site itself? Also, a majority or the urls labeled "not found" in webmaster tools are properly redirected. Do these merely need to be marked as fixed, or is there something else that needs to be fixed like the sitemap structure? I appreciate any and all input. Beyond Indigo
Intermediate & Advanced SEO | | BeyondIndigo1 -
Best way to implement canonical tags on an ecommerce site with many filter options?
What would be the best way to add canonical tags to an ecommerce site with many filter options, for example, http://teacherexpress.scholastic.com? Should I include a canonical tag for all filter options under a category even though the pages don't have the same content? Thanks for reading!
Intermediate & Advanced SEO | | DA20130 -
Emergency duplicate of website due to DNS failure - how to minimise loss of search engine traffic?
Hi, Our client has had a disaster with their domain name registrar, where the DNS settings have been reset and it looks like the registrar won't be able to re-instate the DNS settings for four days time. This is a nightmare for lost business whilst the site and emails are offline. As a fallback, we've setup a copy of the client's website at an alternative domain name so that people can be directed there in the meantime via Facebook posts, etc. Is there anything you would recommend we do in the meantime to minimise the loss of traffic from search engines, and loss of reputation with Google? eg. using Google webmasters to tell Google about the change of address? Thank you.
Intermediate & Advanced SEO | | smaavie0 -
Featured citations vs. regular citations
Do regular citations have the same impact as featured citations for Google Local? or is it the higher a company is isted the more impact it will have on a google local page?
Intermediate & Advanced SEO | | SEODinosaur0 -
When you provide traffic estimates, do you factor in CTR?
There are several studies that show CTR based on position. When a client asks for traffic estimates do you multiply CTR by estimated search volume? Why or why not?
Intermediate & Advanced SEO | | nicole.healthline0 -
Sudden drop in ranks and traffic after migrating community website into main domain
Hi, We recently moved our community website (around 50K web pages) to our main domain. It now resides as a sub-domain on our main website. e.g. Before - we had www.mainwebsite.com and www.communitywebsite.com After - we have www.communitywebsite.mainwebsite.com This change took place on July 19th. After a week, we saw 16% drop in organic traffic to mainwebsite.com. Our ranks on most of the head keywords including brand keywords have dropped. We had created 301 redirects from pages on www.communitywebsite.com before this change was made. Has anybody seen this kind of impact when domains are merged? Should we expect that within 3-4 weeks Google will be able to re-index and re-rank all the pages? Is there anything else we could do to rectify the situation? Any feedback/suggestions are welcome!
Intermediate & Advanced SEO | | Amjath0