Hi Mozers,

How come Moz reports just six 404 errors, whereas Google Search Console reports 250 and Screaming Frog only reports a dozen? It seems to me that these results are all over the place. Shouldn't these reports be more consistent?

I do understand that Search Console includes historical data and that URLs or issues need to be "marked as fixed" in order for them to go away, however, even if I do this, Google ends up reporting far more errors than anything else.

Do 404s reported by Moz and Screaming Frog NOT include external links? It seems to me that this could be partially responsible for the issue.

Also, is there a way to efficiently track the source of the 404s besides clicking on "Linked From" within Search Console 250 times? I was looking for something like this is Moz or SF but no luck.

Any help is appreciated.

Thanksabunch!

Hi SirMax,

Thanks for your input. I appreciate it. We'll add Wordfence to our WordPress toolbox and see if that addresses the issue.

In response to previous posts, thanks to everyone for your input. We were able to apply some filters to remove the bogus bot traffic from the analytics and normalize the data, however, this did not actually resolve the issue and in my eyes is more of a BandAid fix. The evil crawlers are still there, we just can't see them.

Thanks again for all of your input.

Eric

Hi Ruth,

Thanks a bunch for taking the time to respond to my post. Great advice. This is reassuring on a number of levels, however, it doesn't address the underlying issue of how to stop these spam bots in the first place.

We've already started the process of filtering out some of this bogus data. We'll also be integrating some WordPress plugins to see if that helps. That said, if the spam bots are hitting Analytics directly, as opposed to the actual website, WP plugins won't do anything.

Anyway, I appreciate your input and advice. Thanks so much.

Eric

Hello,

I'm hoping one of you search geniuses can help me.

We have a successful client who started seeing a HUGE spike in direct visits as reported by Google Analytics. This traffic now represents approximately 70% of all website traffic. These "direct visits" have a bounce rate of 96%+ and only 1-2 pages/visit. This is skewing our analytics in a big way and rendering them pretty much useless. I suspect this is some sort of crawler activity but we have no access to the server log files to verify this or identify the culprit. The client's site is on a GoDaddy Managed WordPress hosting account.

The way I see it, there are a couple of possibilities.
1.) Our client's competitors are scraping the site on a regular basis to stay on top of site modifications, keyword emphasis, etc. It seems like whenever we make meaningful changes to the site, one of their competitors does a knock-off a few days later. Hmmm.

2.) Our client's competitors have this crawler hitting the site thousands of times a day to raise bounce rates and decrease the average time on site, which could like have an negative impact on SEO. Correct me if I'm wrong but I don't believe Google is going to reward sites with 90% bounce rates, 1-2 pages/visit and an 18 second average time on site.

The bottom line is that we need to identify these bogus "direct visits" and find a way to block them. I've seen several WordPress plugins that claim to help with this but I certainly don't want to block valid crawlers, especially Google, from accessing the site.

If someone out there could please weigh in on this and help us resolve the issue, I'd really appreciate it. Heck, I'll even name my third-born after you.

Thanks for your help.

Eric