Help Blocking Crawlers. Huge Spike in "Direct Visits" with 96% Bounce Rate & Low Pages/Visit.
-
Hello,
I'm hoping one of you search geniuses can help me.
We have a successful client who started seeing a HUGE spike in direct visits as reported by Google Analytics. This traffic now represents approximately 70% of all website traffic. These "direct visits" have a bounce rate of 96%+ and only 1-2 pages/visit. This is skewing our analytics in a big way and rendering them pretty much useless. I suspect this is some sort of crawler activity but we have no access to the server log files to verify this or identify the culprit. The client's site is on a GoDaddy Managed WordPress hosting account.
The way I see it, there are a couple of possibilities.
1.) Our client's competitors are scraping the site on a regular basis to stay on top of site modifications, keyword emphasis, etc. It seems like whenever we make meaningful changes to the site, one of their competitors does a knock-off a few days later. Hmmm.2.) Our client's competitors have this crawler hitting the site thousands of times a day to raise bounce rates and decrease the average time on site, which could like have an negative impact on SEO. Correct me if I'm wrong but I don't believe Google is going to reward sites with 90% bounce rates, 1-2 pages/visit and an 18 second average time on site.
The bottom line is that we need to identify these bogus "direct visits" and find a way to block them. I've seen several WordPress plugins that claim to help with this but I certainly don't want to block valid crawlers, especially Google, from accessing the site.
If someone out there could please weigh in on this and help us resolve the issue, I'd really appreciate it. Heck, I'll even name my third-born after you.
Thanks for your help.
Eric
-
Hi SirMax,
Thanks for your input. I appreciate it. We'll add Wordfence to our WordPress toolbox and see if that addresses the issue.
In response to previous posts, thanks to everyone for your input. We were able to apply some filters to remove the bogus bot traffic from the analytics and normalize the data, however, this did not actually resolve the issue and in my eyes is more of a BandAid fix. The evil crawlers are still there, we just can't see them.
Thanks again for all of your input.
Eric
-
Hostname filtering does not work any more. Unfortunately most of the spammers have adapted and are using your website as hostname.
For the WordPress I use Wordfence plugin( using paid version - not affiliated with them in any shape or form beyond paying for their services). In the advance blocking you can set limits on how fast and how many pages crawlers can request. You can also block by country or ip range. It can also show you live traffic with a lot of details ( a lot more then google analytic - more like server log ). It might not be the complete remedy but it can help.
-
I wish I had an answer for how to stop the bots from hitting your site at all - I don't think a good one exists, as any solutions that wouldn't also block real human traffic to your site are going to be easy for spam bots to get around. I think your best bet is just to do everything you can to keep your data as clean as possible.
-
Hi Ruth,
Thanks a bunch for taking the time to respond to my post. Great advice. This is reassuring on a number of levels, however, it doesn't address the underlying issue of how to stop these spam bots in the first place.
We've already started the process of filtering out some of this bogus data. We'll also be integrating some WordPress plugins to see if that helps. That said, if the spam bots are hitting Analytics directly, as opposed to the actual website, WP plugins won't do anything.
Anyway, I appreciate your input and advice. Thanks so much.
Eric
-
Hi Eric,
A few things to reassure you off the bat:
- For what it's worth, there is a huge, HUGE amount of crawler spam happening in the web today. Every site I work on is being hit hard with false referrals and direct visits. I know Google Analytics is working on a solution to better filter these visits out. So I wouldn't be too concerned that it is something a competitor is doing to your site, specifically - it's more likely that it's been caught up in the general wave of spam crawlers.
- It's important to note that when we talk about Google looking at bounce rate and dwell time as part of ranking your site, those numbers are specifically from clicks through from search - that's data that Google can get without using your private web analytics data as a ranking factor, which they've said repeatedly that they don't and won't do. So a bunch of direct visits with high bounce rates will NOT affect your rankings.
So, it's not dangerous, just annoying. On to how to get that data out of your reports:
- Make sure you're not filtering out spam referrers at a View level - this can cause those visits to incorrectly appear as direct traffic.
- You could set up an Advanced Segment in Google Analytics to filter out direct visits with visit times of, say, under 5 seconds. Some real traffic may get caught in that, but it will get the noise levels down.
- The best way to filter out spam bot traffic, in my opinion, is to set up hostname filtering. Here's a post on Megalytic on how to do that: https://megalytic.com/blog/how-to-filter-out-fake-referrals-and-other-google-analytics-spam. Make sure you've also got an "Unfiltered Data" View so you'll still have historic raw data if you need it.
Hope that helps! Good luck.
-
Check webserver log files, or log visits (ip address, user agent, __utma, __utmz, possibly browser fingerprint, etc...)
Analyzing those you can easily find out if the traffic is from scraping bot or humans.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Increase in Direct Traffic plus Bounce Rate rise for all traffic sources
Hello, I work for an agency and we have seen a big rise in bounce rate for 4 of our clients which happened on the exact same day. This rise on bounce rate is across all traffic sources. We are also seeing a big increase in direct traffic, starting on the same day. Is it possible for bot traffic to affect the bounce rate of all other traffic sources? We have ruled out double reporting in GA but can explain how the bounce rate has increased for all traffic sources. How is this linked to the rise in direct traffic (in some cases as high as 500%)? Thanks
Reporting & Analytics | | jenallen0 -
What is the most effective way of selecting a top keyword per page on a site?
We are creating fresh content for outdated sites and I need to identify the most significant keyword per page for the content developers, What is the best way to do this?
Reporting & Analytics | | Sable_Group0 -
Complete loss of Organic visits within Google Analytics??
Hey guys, I have a pressing query that's really puzzled me! I've just been going through Google Analytics and noticed that there has been a complete drop off of all my organic visitors from May 7. All other traffic mediums are unaffected so there is no issue with the Analytics code recording visits. I haven't received any penalty and my rankings have actually improved in the last month. This is proper bizarre. Any ideas? MjLmL70.png
Reporting & Analytics | | Webrevolve0 -
Organic and direct traffic swap
We moved to a CMS (Webhook) and when we did that organic traffic and direct traffic swapped places. Since we moved it organic traffic is down by about 400 visits and direct traffic is up by 400 visits. I went through the list below and confirmed everything is working. The http refer wasn't being passed for a couple of weeks but the issue was resolved and the organic traffic issue is still ongoing. Is there anything else that may cause this issue? I confirmed the issue isn't one of the below problems. during http to https redirect (or vice versa) the referrer may not be passed incorrect subdomain or cross-domain tracking can strip the referrer. 302 redirects sometimes caused the referrer to be dropped problems with cookies being lost/corrupted. javascript missing from certain entry pages (means any further page view looks like a direct)
Reporting & Analytics | | BT20090 -
Impressions in GWT have dropped to nothing, but my page is still ranking normally
Hello Everyone, I'm seeing a strange issue. On the 22nd of this month Webmasters tools started showing 6 impressions per day down from hundreds or thousands. I thought I was hit with a huge penalty for my keywords but they are still ranking where they have for the past month or two on Google. In analytics my organic traffic is stable. It just seems to be GWT showing the massive drop. My domain is: http://Patchofland.com Any Thoughts? Thanks in advance!
Reporting & Analytics | | PatchofLand0 -
Google Analytics & Omniture Discrepancies
I am seeing a significant difference between my traffic numbers in Google Analytics and Omniture (Omniture has significantly more). I do not expect them to report exactly the same numbers but these are just too far off. Any idea why that is, or which one I should trust more? Thanks!
Reporting & Analytics | | emediaSEO0 -
Webmaster Tools shows 0 visits, Analytics shows 1000s?
Webmaster Tools reported a "Big traffic change for top URL" for one of our sites about three and a half weeks ago. WT traffic is 0 since that date. However, our analytics data shows no significant decline in traffic. The WT alert was received on Nov. 16. There was a significant dip in GA visits Nov. 22 but it recovered and I'm seeing no other indication that this is a penalty. What could be going on?
Reporting & Analytics | | IMM0 -
Keep multiple Google Analytic / Webmaster accounts seperate?
Hi, I have a paranoid suspicion that putting all my websites on one Webmaster / Analytic account will mean any links between my sites won't carry any weight, and Google might consider similar content much more readily if 2 of my sites are in the same market. So I have about 4 different accounts with my main sites split across them. Am I being over paranoid? Do any of you have any problem with all your sites being on the same account?
Reporting & Analytics | | SpecialCase0