Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Regular Expressions for Filtering BOT Traffic?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Regular Expressions for Filtering BOT Traffic?

    Intermediate & Advanced SEO
    3
    17
    3427
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • AWCthreads
      AWCthreads last edited by

      I've set up a filter to remove bot traffic from Analytics. I relied on regular expressions posted in an article that eliminates what appears to be most of them.

      However, there are other bots I would like to filter but I'm having a hard time determining the regular expressions for them.

      How do I determine what the regular expression is for additional bots so I can apply them to the filter?

      I read an Analytics "how to" but its over my head and I'm hoping for some "dumbed down" guidance. 🙂

      1 Reply Last reply Reply Quote 1
      • Chris_CM
        Chris_CM @AWCthreads last edited by

        No problem, feel free to reach out if you have any other RegEx related questions.

        Regards,

        Chris

        1 Reply Last reply Reply Quote 1
        • AWCthreads
          AWCthreads @Chris_CM last edited by

          I will definitely do that for Rackspace bots, Chris.

          Thank you for taking the time to walk me through this and tweak my filter.

          I'll give the site you posted a visit.

          Chris_CM 1 Reply Last reply Reply Quote 0
          • Chris_CM
            Chris_CM @AWCthreads last edited by

            If you copy and paste my RegEx, it will filter out the rackspace bots.  If you want to learn more about Regular Expressions, here is a site that explains them very well, though it may not be quite kindergarten speak.

            AWCthreads 1 Reply Last reply Reply Quote 1
            • AWCthreads
              AWCthreads @Chris_CM last edited by

              Crap.

              Well, I guess the vernacular is what I need to know.

              Knowing what to put where is the trick isn't it? Is there a dummies guide somewhere that spells this out in kindergarten speak?

              I could really see myself botching this filtering business.

              Chris_CM 1 Reply Last reply Reply Quote 0
              • Chris_CM
                Chris_CM last edited by

                Not unless there's a . after the word servers in the name.  The . is escaping the . at the end of stumbleupon inc.

                AWCthreads 1 Reply Last reply Reply Quote 1
                • AWCthreads
                  AWCthreads @Chris_CM last edited by

                  Does it need the . before the )

                  1 Reply Last reply Reply Quote 0
                  • Chris_CM
                    Chris_CM @AWCthreads last edited by

                    Ok, try this:

                    ^(microsoft corp|inktomi corporation|yahoo! inc.|google inc.|stumbleupon inc.|rackspace cloud servers)$|gomez

                    Just added rackspace as another match, it should work if the name is exactly right.

                    Hope this helps,

                    Chris

                    AWCthreads 1 Reply Last reply Reply Quote 1
                    • SErOb
                      SErOb @Chris_CM last edited by

                      Agreed! That's why I suggest using it in combination with the variables you mentioned above.

                      1 Reply Last reply Reply Quote 0
                      • AWCthreads
                        AWCthreads @Chris_CM last edited by

                        rackspace cloud servers

                        Maybe my problem is I'm not looking in the right place.

                        I'm in audience>technology>network and the column shows "service provider."

                        Chris_CM 1 Reply Last reply Reply Quote 0
                        • Chris_CM
                          Chris_CM @AWCthreads last edited by

                          How is it titled in the ISP report exactly?

                          AWCthreads 1 Reply Last reply Reply Quote 0
                          • AWCthreads
                            AWCthreads @Chris_CM last edited by

                            For example,

                            Since I implemented the filter four days ago, rackspace cloud servers have visited my site 848 times, , visited 1 page each time, spent 0 seconds on the page and bounced 100% of the time.

                            What is the reg expression for rackspace?

                            Chris_CM 1 Reply Last reply Reply Quote 0
                            • Chris_CM
                              Chris_CM @SErOb last edited by

                              Time on page can be a tricky one because sometimes actual visits can record 00:00:00 due to the way it is measured.  I'd recommend using other factors like the ones I mentioned above.

                              SErOb 1 Reply Last reply Reply Quote 0
                              • SErOb
                                SErOb @Chris_CM last edited by

                                "...a combination of operating system, location, and some other factors can do the trick."

                                Yep, combined with those, look for "Avg. Time on Page = 00:00:00"

                                Chris_CM 1 Reply Last reply Reply Quote 1
                                • Chris_CM
                                  Chris_CM last edited by

                                  Ok, can you provide some information on the bots that are getting through this that you want to sort out? If they are able to be filtered through the ISP organization as the ones in your current RegEx, you can simply add them to the list: (microsoft corp| ...       ... |stumbleupon inc.|ispnamefromyourbots|ispname2|etc.)$|gomez

                                  Otherwise, you might need to get creative and find another way to isolate them (a combination of operating system, location, and some other factors can do the trick).  When adding to the list, make sure to escape special characters like . or / by using a \ before them, or else your RegEx will fail.

                                  SErOb AWCthreads 2 Replies Last reply Reply Quote 1
                                  • AWCthreads
                                    AWCthreads @Chris_CM last edited by

                                    Sure. Here's the post for filtering the bots.

                                    Here's the reg x posted:  ^(microsoft corp|inktomi corporation|yahoo! inc.|google inc.|stumbleupon inc.)$|gomez

                                    1 Reply Last reply Reply Quote 0
                                    • Chris_CM
                                      Chris_CM last edited by

                                      If you give me an idea of how you are isolating the bots I might be able to help come up with a RegEx for you.  What is the RegEx you have in place to sort out the other bots?

                                      Regards,

                                      Chris

                                      AWCthreads 1 Reply Last reply Reply Quote 1
                                      • 1 / 1
                                      • First post
                                        Last post

                                      Got a burning SEO question?

                                      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                      Start my free trial


                                      Browse Questions

                                      Explore more categories

                                      • Moz Tools

                                        Chat with the community about the Moz tools.

                                      • SEO Tactics

                                        Discuss the SEO process with fellow marketers

                                      • Community

                                        Discuss industry events, jobs, and news!

                                      • Digital Marketing

                                        Chat about tactics outside of SEO

                                      • Research & Trends

                                        Dive into research and trends in the search industry.

                                      • Support

                                        Connect on product support and feature requests.

                                      • See all categories

                                      Related Questions

                                      • inhouseninja

                                        Huge Drop in Direct Traffic in G4

                                        google analytics direct traffic

                                        Our direct traffic dropped 50% in October. Is anyone else seeing a drop in direct traffic in October in G4? It hasn't shifted to another source or unassigned it's just gone. Has anyone else experienced this and what might be the reasons?

                                        Intermediate & Advanced SEO | | inhouseninja
                                        1
                                      • Dennis1992038

                                        Huge organic traffic drom after a perfect domain migration. What to do?

                                        Hi, I already asked the question on different places. But so far nobody could help me. 
                                        Hope someone can help me out. If possible.
                                        I migrated my website https://vihara.nl to https://meditatieinstituut.nl and lost about 80% traffic (see printscreens). It's over more than a month ago now and there is no sign of getting it back up. Maybe there is nothing to do and
                                        1. I have to be patient and traffic comes back in a few months. 
                                        or
                                        2. There is nothing to do and I've lost everything I've build up in the last years. Start over again to get the rankings back.
                                        or maybe, maybe
                                        3. I just forgot something that I still need to do to get the rankings back up. Or there is something I did not think of... This is done: The website is migrated 1 on 1. No changes in content, url, code, etc. Everything is exactly the same as on the previous domain. 301 redirects whole domain (via htaccess a bulk redirect). All the old pages, without exceptions, lead to the exact new page. The new domain is running from CDN (Cloudflare) with the same settings as the previous domain. SSL is installed in the exact same way. Domain migration set up in Search console (working). Uploaded new sitemap (working). Updated internal links. Changed the most important external links (where I could get contact after reaching out) In meanwhile received some new external links and also posted new content Anybody knows what to do? Or do I just have to be more patient and will it come back in a few months by itself? Looking forward to suggetions. Thanks! Gerjan Migratie-Meditatie-Instituut-2048x786.jpg verloop-sinds-de-start-2048x355.jpg

                                        Intermediate & Advanced SEO | | Dennis1992038
                                        0
                                      • Kingalan1

                                        Organic Traffic Drop of 90% After Domain Migration

                                        We moved our domain is http://www.nyc-officespace-leader.com on April 4th. It was migrated to https://www.metro-manhattan.com Google Search Console continues to show  about 420of URLs indexed for the old "NYC" domain. This number has not dropped on Search Console. Don't understand why Google has not de-indexed the old site. 
                                        For the new "Metro" domain only 114 pages are being shown as valid. Our search volume has dropped from about 85 visits a day to 12 per day. 390 URLs appear as "crawled- currently not indexed". Please note that the migrated content is identical. Nothing at all changed. All re-directs were implemented properly. Also, at the time of the migration we filed a disavow for about 200 spammy links. This disavow file was entered for the old domain and the new one as well. Any ideas as to how to trouble shoot this would be much appreciated!!! This has not been very good for business.

                                        Intermediate & Advanced SEO | | Kingalan1
                                        0
                                      • Eric_OWPP

                                        Huge Spike in Organic/Direct traffic from Mexico

                                        So here's my situation: My company's website usually receives around 80 organic visits/month and 50 direct visits/month from Mexico. However, in July we saw a small uptick to around 170 for each and then in the last 7 days we are in the middle of a massive spike which has put us up to 1400 visits for organic and 820 visits for direct in August. The traffic spike continues as we are almost up to 500 visits just today! Things to know: The visitors are purchasing from our store, staying on our site, browsing around, basically acting like real traffic. I was unable to identify any new links, press, and we did not do any specific Mexico optimization (spanish keywords). We sell a ball and it is called The One World Futbol, but it's always been called a futbol before so nothing new here. our website is www.oneworldplayproject.com. Everyone coming organically is searching our name, not keywords. We updated our shopping cart a few days before the massive traffic spike and significantly lowered the cost to ship to Mexico. Our Latin America director went to Mexico to work there for a month a few days before the spike and sent out a bunch of emails, texts, phone calls, what's app notifications to his large network. From what I am told by others here he has a vast network throughout Mexico, Central America and South America. We have also seen large traffic increases in other Latin American countries during this same time period just nothing like Mexico. We just hired an awesome social media coordinator who is extremely focused and is implementing a kick-ass social strategy We launched a branding campaign called #MakeLifePlayFull with press releases and ad spend behind it. PHEW! That was a lot of info for you to digest. So on the surface this seems like great news. BUT I want to understand WHY this is happening. Could it really just be the combination of all these things listed above or is it just a combination of our connected guy being in Mexico with better shipping costs? Why is it mainly happening in Mexico? Why is it so sustained? I suspect that if it is from our guy it would drop off quickly. Any thoughts on what to look at? I'm stumped.

                                        Intermediate & Advanced SEO | | Eric_OWPP
                                        0
                                      • digitalcrc

                                        Best-practice URL structures with multiple filter combinations

                                        Hello, We're putting together a large piece of content that will have some interactive filtering elements. There are two types of filters, topics and object types. The architecture under the hood constrains us so that everything needs to be in URL parameters. If someone selects a single filter, this can look pretty clean: www.domain.com/project?topic=firstTopic
                                        or
                                        www.domain.com/project?object=typeOne The problems arise when people select multiple topics, potentially across two different filter types: www.domain.com/project?topic=firstTopic-secondTopic-thirdTopic&object=typeOne-typeTwo I've raised concerns around the structure in general, but it seems to be too late at this point so now I'm scratching my head thinking of how best to get these indexed. I have two main concerns: A ton of near-duplicate content and hundreds of URLs being created and indexed with various filter combinations added Over-reacting to the first point above and over-canonicalizing/no-indexing combination pages to the detriment of the content as a whole Would the best approach be to index each single topic filter individually, and canonicalize any combinations to the 'view all' page? I don't have much experience with e-commerce SEO (which this problem seems to have the most in common with) so any advice is greatly appreciated. Thanks!

                                        Intermediate & Advanced SEO | | digitalcrc
                                        0
                                      • HB17

                                        Spike then Drop in Direct Traffic?

                                        We've been doing some SEO work over the last few weeks and earlier this week we saw a large spike in traffic. Yay we all thought, but then yesterday the traffic levels returned to pre-celebratory levels. I've been doing some digging to try and find out what was different Monday and Tuesday this week. Mondays are usually big traffic days for us anyway, but this week was by far the biggest, and Tuesday was even higher still, our best day ever. After some poking, I found that the direct traffic followed the same pattern as our overall traffic levels (image attached). The first spike coincides with an email we sent out that day, but the later spike we just don't know where it came from? I understand loosely that direct isn't easily traceable, but can anyone help us understand more about this second spike? Thanks! ayqL2wi

                                        Intermediate & Advanced SEO | | HB17
                                        0
                                      • DA2013

                                        Best way to implement canonical tags on an ecommerce site with many filter options?

                                        What would be the best way to add canonical tags to an ecommerce site with many filter options, for example, http://teacherexpress.scholastic.com? Should I include a canonical tag for all filter options under a category even though the pages don't have the same content? Thanks for reading!

                                        Intermediate & Advanced SEO | | DA2013
                                        0
                                      • footsteps

                                        How to prevent Google from crawling our product filter?

                                        Hi All, We have a crawler problem on one of our sites www.sneakerskoopjeonline.nl. On this site, visitors can specify criteria to filter available products. These filters are passed as http/get arguments. The number of possible filter urls is virtually limitless. In order to prevent duplicate content, or an insane amount of pages in the search indices, our software automatically adds noindex, nofollow and noarchive directives to these filter result pages. However, we’re unable to explain to crawlers (Google in particular) to ignore these urls. We’ve already changed the on page filter html to javascript, hoping this would cause the crawler to ignore it. However, it seems that Googlebot executes the javascript and crawls the generated urls anyway. What can we do to prevent Google from crawling all the filter options? Thanks in advance for the help. Kind regards, Gerwin

                                        Intermediate & Advanced SEO | | footsteps
                                        0

                                      Get started with Moz Pro!

                                      Unlock the power of advanced SEO tools and data-driven insights.

                                      Start my free trial
                                      Products
                                      • Moz Pro
                                      • Moz Local
                                      • Moz API
                                      • Moz Data
                                      • STAT
                                      • Product Updates
                                      Moz Solutions
                                      • SMB Solutions
                                      • Agency Solutions
                                      • Enterprise Solutions
                                      Free SEO Tools
                                      • Domain Authority Checker
                                      • Link Explorer
                                      • Keyword Explorer
                                      • Competitive Research
                                      • Brand Authority Checker
                                      • Local Citation Checker
                                      • MozBar Extension
                                      • MozCast
                                      Resources
                                      • Blog
                                      • SEO Learning Center
                                      • Help Hub
                                      • Beginner's Guide to SEO
                                      • How-to Guides
                                      • Moz Academy
                                      • API Docs
                                      About Moz
                                      • About
                                      • Team
                                      • Careers
                                      • Contact
                                      Why Moz
                                      • Case Studies
                                      • Testimonials
                                      Get Involved
                                      • Become an Affiliate
                                      • MozCon
                                      • Webinars
                                      • Practical Marketer Series
                                      • MozPod
                                      Connect with us

                                      Contact the Help team

                                      Join our newsletter
                                      Moz logo
                                      © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                      • Accessibility
                                      • Terms of Use
                                      • Privacy

                                      Looks like your connection to Moz was lost, please wait while we try to reconnect.