undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Intermediate & Advanced SEO
  4. Regular Expressions for Filtering BOT Traffic?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Regular Expressions for Filtering BOT Traffic?

Intermediate & Advanced SEO
3
17
3.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • AWCthreads
    AWCthreads last edited by Sep 30, 2013, 12:15 PM

    I've set up a filter to remove bot traffic from Analytics. I relied on regular expressions posted in an article that eliminates what appears to be most of them.

    However, there are other bots I would like to filter but I'm having a hard time determining the regular expressions for them.

    How do I determine what the regular expression is for additional bots so I can apply them to the filter?

    I read an Analytics "how to" but its over my head and I'm hoping for some "dumbed down" guidance. 🙂

    1 Reply Last reply Reply Quote 1
    • Chris_CM
      Chris_CM @AWCthreads last edited by Sep 30, 2013, 6:14 PM Sep 30, 2013, 6:14 PM

      No problem, feel free to reach out if you have any other RegEx related questions.

      Regards,

      Chris

      1 Reply Last reply Reply Quote 1
      • AWCthreads
        AWCthreads @Chris_CM last edited by Sep 30, 2013, 6:06 PM Sep 30, 2013, 6:06 PM

        I will definitely do that for Rackspace bots, Chris.

        Thank you for taking the time to walk me through this and tweak my filter.

        I'll give the site you posted a visit.

        Chris_CM 1 Reply Last reply Sep 30, 2013, 6:14 PM Reply Quote 0
        • Chris_CM
          Chris_CM @AWCthreads last edited by Sep 30, 2013, 6:07 PM Sep 30, 2013, 6:03 PM

          If you copy and paste my RegEx, it will filter out the rackspace bots.  If you want to learn more about Regular Expressions, here is a site that explains them very well, though it may not be quite kindergarten speak.

          AWCthreads 1 Reply Last reply Sep 30, 2013, 6:06 PM Reply Quote 1
          • AWCthreads
            AWCthreads @Chris_CM last edited by Sep 30, 2013, 5:56 PM Sep 30, 2013, 5:56 PM

            Crap.

            Well, I guess the vernacular is what I need to know.

            Knowing what to put where is the trick isn't it? Is there a dummies guide somewhere that spells this out in kindergarten speak?

            I could really see myself botching this filtering business.

            Chris_CM 1 Reply Last reply Sep 30, 2013, 6:03 PM Reply Quote 0
            • Chris_CM
              Chris_CM last edited by Sep 30, 2013, 6:08 PM Sep 30, 2013, 5:51 PM

              Not unless there's a . after the word servers in the name.  The . is escaping the . at the end of stumbleupon inc.

              AWCthreads 1 Reply Last reply Sep 30, 2013, 5:56 PM Reply Quote 1
              • AWCthreads
                AWCthreads @Chris_CM last edited by Sep 30, 2013, 5:49 PM Sep 30, 2013, 5:49 PM

                Does it need the . before the )

                1 Reply Last reply Reply Quote 0
                • Chris_CM
                  Chris_CM @AWCthreads last edited by Sep 30, 2013, 6:07 PM Sep 30, 2013, 5:45 PM

                  Ok, try this:

                  ^(microsoft corp|inktomi corporation|yahoo! inc.|google inc.|stumbleupon inc.|rackspace cloud servers)$|gomez

                  Just added rackspace as another match, it should work if the name is exactly right.

                  Hope this helps,

                  Chris

                  AWCthreads 1 Reply Last reply Sep 30, 2013, 5:49 PM Reply Quote 1
                  • SErOb
                    SErOb @Chris_CM last edited by Sep 30, 2013, 5:45 PM Sep 30, 2013, 5:45 PM

                    Agreed! That's why I suggest using it in combination with the variables you mentioned above.

                    1 Reply Last reply Reply Quote 0
                    • AWCthreads
                      AWCthreads @Chris_CM last edited by Sep 30, 2013, 5:46 PM Sep 30, 2013, 5:42 PM

                      rackspace cloud servers

                      Maybe my problem is I'm not looking in the right place.

                      I'm in audience>technology>network and the column shows "service provider."

                      Chris_CM 1 Reply Last reply Sep 30, 2013, 5:45 PM Reply Quote 0
                      • Chris_CM
                        Chris_CM @AWCthreads last edited by Sep 30, 2013, 5:40 PM Sep 30, 2013, 5:40 PM

                        How is it titled in the ISP report exactly?

                        AWCthreads 1 Reply Last reply Sep 30, 2013, 5:42 PM Reply Quote 0
                        • AWCthreads
                          AWCthreads @Chris_CM last edited by Sep 30, 2013, 5:40 PM Sep 30, 2013, 5:38 PM

                          For example,

                          Since I implemented the filter four days ago, rackspace cloud servers have visited my site 848 times, , visited 1 page each time, spent 0 seconds on the page and bounced 100% of the time.

                          What is the reg expression for rackspace?

                          Chris_CM 1 Reply Last reply Sep 30, 2013, 5:40 PM Reply Quote 0
                          • Chris_CM
                            Chris_CM @SErOb last edited by Sep 30, 2013, 5:38 PM Sep 30, 2013, 5:38 PM

                            Time on page can be a tricky one because sometimes actual visits can record 00:00:00 due to the way it is measured.  I'd recommend using other factors like the ones I mentioned above.

                            SErOb 1 Reply Last reply Sep 30, 2013, 5:45 PM Reply Quote 0
                            • SErOb
                              SErOb @Chris_CM last edited by Sep 30, 2013, 5:35 PM Sep 30, 2013, 5:35 PM

                              "...a combination of operating system, location, and some other factors can do the trick."

                              Yep, combined with those, look for "Avg. Time on Page = 00:00:00"

                              Chris_CM 1 Reply Last reply Sep 30, 2013, 5:38 PM Reply Quote 1
                              • Chris_CM
                                Chris_CM last edited by Sep 30, 2013, 5:26 PM Sep 30, 2013, 5:22 PM

                                Ok, can you provide some information on the bots that are getting through this that you want to sort out? If they are able to be filtered through the ISP organization as the ones in your current RegEx, you can simply add them to the list: (microsoft corp| ...       ... |stumbleupon inc.|ispnamefromyourbots|ispname2|etc.)$|gomez

                                Otherwise, you might need to get creative and find another way to isolate them (a combination of operating system, location, and some other factors can do the trick).  When adding to the list, make sure to escape special characters like . or / by using a \ before them, or else your RegEx will fail.

                                SErOb AWCthreads 2 Replies Last reply Sep 30, 2013, 5:38 PM Reply Quote 1
                                • AWCthreads
                                  AWCthreads @Chris_CM last edited by Sep 30, 2013, 5:02 PM Sep 30, 2013, 5:02 PM

                                  Sure. Here's the post for filtering the bots.

                                  Here's the reg x posted:  ^(microsoft corp|inktomi corporation|yahoo! inc.|google inc.|stumbleupon inc.)$|gomez

                                  1 Reply Last reply Reply Quote 0
                                  • Chris_CM
                                    Chris_CM last edited by Sep 30, 2013, 4:04 PM Sep 30, 2013, 4:04 PM

                                    If you give me an idea of how you are isolating the bots I might be able to help come up with a RegEx for you.  What is the RegEx you have in place to sort out the other bots?

                                    Regards,

                                    Chris

                                    AWCthreads 1 Reply Last reply Sep 30, 2013, 5:02 PM Reply Quote 1
                                    • 1 / 1
                                    1 out of 17
                                    • First post
                                      1/17
                                      Last post

                                    Got a burning SEO question?

                                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                    Start my free trial


                                    Browse Questions

                                    Explore more categories

                                    • Moz Tools

                                      Chat with the community about the Moz tools.

                                    • SEO Tactics

                                      Discuss the SEO process with fellow marketers

                                    • Community

                                      Discuss industry events, jobs, and news!

                                    • Digital Marketing

                                      Chat about tactics outside of SEO

                                    • Research & Trends

                                      Dive into research and trends in the search industry.

                                    • Support

                                      Connect on product support and feature requests.

                                    • See all categories

                                    Related Questions

                                    • Dennis1992038

                                      Huge organic traffic drom after a perfect domain migration. What to do?

                                      Hi, I already asked the question on different places. But so far nobody could help me. 
                                      Hope someone can help me out. If possible.
                                      I migrated my website https://vihara.nl to https://meditatieinstituut.nl and lost about 80% traffic (see printscreens). It's over more than a month ago now and there is no sign of getting it back up. Maybe there is nothing to do and
                                      1. I have to be patient and traffic comes back in a few months. 
                                      or
                                      2. There is nothing to do and I've lost everything I've build up in the last years. Start over again to get the rankings back.
                                      or maybe, maybe
                                      3. I just forgot something that I still need to do to get the rankings back up. Or there is something I did not think of... This is done: The website is migrated 1 on 1. No changes in content, url, code, etc. Everything is exactly the same as on the previous domain. 301 redirects whole domain (via htaccess a bulk redirect). All the old pages, without exceptions, lead to the exact new page. The new domain is running from CDN (Cloudflare) with the same settings as the previous domain. SSL is installed in the exact same way. Domain migration set up in Search console (working). Uploaded new sitemap (working). Updated internal links. Changed the most important external links (where I could get contact after reaching out) In meanwhile received some new external links and also posted new content Anybody knows what to do? Or do I just have to be more patient and will it come back in a few months by itself? Looking forward to suggetions. Thanks! Gerjan Migratie-Meditatie-Instituut-2048x786.jpg verloop-sinds-de-start-2048x355.jpg

                                      Intermediate & Advanced SEO | Jun 12, 2020, 6:06 PM | Dennis1992038
                                      0
                                    • ScottOlson

                                      Can Google Crawl AJAX filters?

                                      Can Google crawl and render pages within Ajax Filters?

                                      Intermediate & Advanced SEO | Jun 29, 2015, 2:31 PM | ScottOlson
                                      0
                                    • morg45454

                                      Robots.txt - Do I block Bots from crawling the non-www version if I use www.site.com ?

                                      my site uses is set up at http://www.site.com I have my site redirected from non- www to the www in htacess file. My question is... what should my robots.txt file look like for the non-www site? Do you block robots from crawling the site like this? Or do you leave it blank? User-agent: * Disallow: / Sitemap: http://www.morganlindsayphotography.com/sitemap.xml Sitemap: http://www.morganlindsayphotography.com/video-sitemap.xml

                                      Intermediate & Advanced SEO | Jun 17, 2015, 5:48 PM | morg45454
                                      0
                                    • HB17

                                      Spike then Drop in Direct Traffic?

                                      We've been doing some SEO work over the last few weeks and earlier this week we saw a large spike in traffic. Yay we all thought, but then yesterday the traffic levels returned to pre-celebratory levels. I've been doing some digging to try and find out what was different Monday and Tuesday this week. Mondays are usually big traffic days for us anyway, but this week was by far the biggest, and Tuesday was even higher still, our best day ever. After some poking, I found that the direct traffic followed the same pattern as our overall traffic levels (image attached). The first spike coincides with an email we sent out that day, but the later spike we just don't know where it came from? I understand loosely that direct isn't easily traceable, but can anyone help us understand more about this second spike? Thanks! ayqL2wi

                                      Intermediate & Advanced SEO | May 24, 2015, 9:47 PM | HB17
                                      0
                                    • damienthivolle

                                      Subdomains vs directories on existing website with good search traffic

                                      Hello everyone, I operate a website called Icy Veins (www.icy-veins.com), which gives gaming advice for World of Warcraft and Hearthstone, two titles from Blizzard Entertainment. Up until recently, we had articles for both games on the main subdomain (www.icy-veins.com), without a directory structure. The articles for World of Warcraft ended in -wow and those for Hearthstone ended in -hearthstone and that was it. We are planning to cover more games from Blizzard entertainment soon, so we hired a SEO consultant to figure out whether we should use directories (www.icy-veins.com/wow/, www.icy-veins.com/hearthstone/, etc.) or subdomains (www.icy-veins.com, wow.icy-veins.com, hearthstone.icy-veins.com). For a number of reason, the consultant was adamant that subdomains was the way to go. So, I implemented subdomains and I have 301-redirects from all the old URLs to the new ones, and after 2 weeks, the amount of search traffic we get has been slowly decreasing, as the new URLs were getting index. Now, we are getting about 20%-25% less search traffic. For example, the week before the subdomains went live we received 900,000 visits from search engines (11-17 May). This week, we only received 700,000 visits. All our new URLs are indexed, but they rank slightly lower than the old URLs used to, so I was wondering if this was something that was to be expected and that will improve in time or if I should just go for subdomains. Thank you in advance.

                                      Intermediate & Advanced SEO | Jun 4, 2014, 9:32 AM | damienthivolle
                                      0
                                    • DougRoberts

                                      Can a large fluctuation of links cause traffic loss?

                                      I've been asked to look at a site that has lost 70/80% if their search traffic. This happened suddenly around the 17th April. Traffic dropped off over a couple of days and then flat-lined over the next couple of weeks. The screenshot attached, shows the impressions/clicks reported in GWT. When I investigated I found: There had been no changes/updates to the site in question There were no messages in GWT indicating a manual penalty The number of pages indexed shows no significant change There are no particular trends in keywords/queries affected (they all were.) I did discover that ahrefs.com showed that a large number of links were reported lost on the 17th April. (17k links from 1 domain). These links reappeared around the 26th/27th April. But traffic shows no sign of any recovery. The links in question were from a single development server (that shouldn't have been indexed in the first place, but that's another matter.) Is it possible that these links were, maybe artificially, boosting the authority of the affected site? Has the sudden fluctuation in such a large number of links caused the site to trip an algorithmic penalty (penguin?) Without going into too much detail as I'm bound by client confidentiality - The affected site is really a large database and the links pointing to it are generated by a half dozen or so article based sister sites based on how the articles are tagged. The links point to dynamically generated content based on the url. The site does provide a useful/valuable service/purpose - it's not trying to "game the system" in order to rank. That doesn't mean to say that it hasn't been performing better in search than it should have been. This means that the affected site has ~900,000 links pointing to is that are the names of different "entities". Any thoughts/insights would be appreciated. I've expresses a pessimistic outlook to the client, but as you can imaging they are confused and concerned. LVSceCN.png

                                      Intermediate & Advanced SEO | Jun 4, 2014, 3:44 AM | DougRoberts
                                      0
                                    • JakubH

                                      Lost 86% of traffic after moving old static site to WordPress

                                      I hired a company to convert an old static website www.rawfoodexplained.com with about 1200 pages of content to WordPress. Four days after launch it lost almost 90% of traffic. It was getting over 60,000 uniques while nobody touched the site for several years. It’s been 21 days since the WordPress launch. I read a lot of stuff prior to moving it (including Moz's case study) and I was expecting to lose in short term 30% of traffic max… I don’t understand what is wrong. The internal link structure is the same, every url is 301 to the same url only without[dot]html (ie www.rawfoodexplained.com/science.html is 301′s to http://www.rawfoodexplained.com/science/ ), it’s added to Google Webmaster tool and Google indexed the new pages… Any ideas what could be possible wrong? I do understand the website is not optimized (meta descriptions etc, but it wasn't before either) .... Do you think putting back the old site would recover the traffic? I would appreciate any thoughts Thank you

                                      Intermediate & Advanced SEO | Oct 20, 2013, 12:01 PM | JakubH
                                      0
                                    • EEE3

                                      Subdomain Blog Sitemap link - Add it to regular domain?

                                      Example of setup:
                                      www.fancydomain.com
                                      blog.fancydomain.com Because of certain limitations, I'm told we can't put our blogs at the subdirectory level, so we are hosting our blogs at the subdomain level (blog.fancydomain.com). I've been asked to incorporate the blog's sitemap link on the regular domain, or even in the regular domain's sitemap. 1. Putting the a link to blog.fancydomain.com/sitemap_index.xml in the www.fancydomain.com/sitemap.xml -- isn't this against sitemap.org protocol? 2. Is there even a reason to do this? We do have a link to the blog's home page from the www.fancydomain.com navigation, and the blog is set up with its sitemap and link to the sitemap in the footer. 3. What about just including a text link "Blog Sitemap" (linking to blog.fancydomain.com/sitemap_index.html) in the footer of the www.fancydomain.com (adjacent to the text link "Sitemap" which already exists for the www.fancydomain.com's sitemap. Just trying to make sense of this, and figure out why or if it should be done. Thanks!

                                      Intermediate & Advanced SEO | Jun 14, 2013, 3:21 PM | EEE3
                                      0

                                    Get started with Moz Pro!

                                    Unlock the power of advanced SEO tools and data-driven insights.

                                    Start my free trial
                                    Products
                                    • Moz Pro
                                    • Moz Local
                                    • Moz API
                                    • Moz Data
                                    • STAT
                                    • Product Updates
                                    Moz Solutions
                                    • SMB Solutions
                                    • Agency Solutions
                                    • Enterprise Solutions
                                    Free SEO Tools
                                    • Domain Authority Checker
                                    • Link Explorer
                                    • Keyword Explorer
                                    • Competitive Research
                                    • Brand Authority Checker
                                    • Local Citation Checker
                                    • MozBar Extension
                                    • MozCast
                                    Resources
                                    • Blog
                                    • SEO Learning Center
                                    • Help Hub
                                    • Beginner's Guide to SEO
                                    • How-to Guides
                                    • Moz Academy
                                    • API Docs
                                    About Moz
                                    • About
                                    • Team
                                    • Careers
                                    • Contact
                                    Why Moz
                                    • Case Studies
                                    • Testimonials
                                    Get Involved
                                    • Become an Affiliate
                                    • MozCon
                                    • Webinars
                                    • Practical Marketer Series
                                    • MozPod
                                    Connect with us

                                    Contact the Help team

                                    Join our newsletter
                                    Moz logo
                                    © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                    • Accessibility
                                    • Terms of Use
                                    • Privacy

                                    Looks like your connection to Moz was lost, please wait while we try to reconnect.