Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
-
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.
To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.
Context that might be helpful:
- I did check technical settings, and the sites are crawlable.
- The sites do appear in search but seem to be losing organic search traffic.
Thanks for any help you can provide!
-
Hey David,
I thought I'd jump in here, as it's our tool
We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -
https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots
You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/
Cheers.
Dan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
US domain pages showing up in Google UK SERP
Hi, Our website which was predominantly for UK market was setup with a .com extension and only two years ago other domains were added - US (.us) , IE (.ie), EU (.eu) & AU (.com.au) Last year in July, we noticed that few .us domain urls were showing up in UK SERPs and we realized the sitemap for .us site was incorrectly referring to UK (.com) so we corrected that and the .us domain urls stopped appearing in the SERP. Not sure if this actually fixed the issue or was such coincidental. However in last couple of weeks more than 3 .us domain urls are showing for each brand search made on Google UK and sometimes it replaces the .com results all together. I have double checked the PA for US pages, they are far below the UK ones. Has anyone noticed similar behaviour &/or could anyone please help me troubleshoot this issue? Thanks in advance, R
Algorithm Updates | | RaksG0 -
Why do we have so many pages scanned by bots (over 250,000) and our biggest competitors have about 70,000? Seems like something is very wrong.
We are trying to figure out why last year we had a huge (80%) and sudden (within two days) drop in our google searches. The only "outlier" in our site that we can find is a huge number of pages reported in MOZ as scanned by search engines. Is this a problem? How did we get so many pages reported? What can we do to bring the number of searched pages back to a "normal" level? BT
Algorithm Updates | | achituv0 -
Two months - No Articles or Post Published in our blog. Moz shows less organic traffic.
Two months - No Articles or Post Published in our blog. Moz shows less organic traffic. i know i could not write - i was sick. organic search and keyword also. total pageviews dropped. DA increased by +3 and then -1 in last update. What should i do.
Algorithm Updates | | Esaky0 -
Authorship Photo Not showing in for last 6 months now
Hi - Early last year we activated Authorship Photo which remained in SERP till Nov'12... However, post that the Authorship Photo is not showing, neither the schema rating tags are showing in Search engines We tried a lot by changing lot of hits and tries, still to no avail.. We have written to google twice for it (they had a link for which author whose photo now showing after everything is right can send site links )- but to no avail.... The Rich snippet shows Authorship Photo, Schema rating. Even, Google custom search on our own site own page showing it... However, this does not translated into any of these shown in actual Google Search Engines. Our Site is :- http://www.mycarhelpline.com/ Sample Links of rich snippet http://www.google.com/webmasters/tools/richsnippets?q=http%3A%2F%2Fwww.mycarhelpline.com%2Findex.php%3Foption%3Dcom_easyblog%26view%3Dentry%26id%3D94%26Itemid%3D91&html= http://www.google.com/webmasters/tools/richsnippets?q=http%3A%2F%2Fwww.mycarhelpline.com%2Findex.php%3Foption%3Dcom_latestnews%26view%3Ddetail%26n_id%3D467%26Itemid%3D10&html= Even google custom search showing schema rating tags for searched keywords like :- Ford Ecosport, Tata Nano Diesel .... However, on actual search the schema tags are now shown Can anyone suggest - what am i missing, actually lost on this...... Worse - our SERP are somehow also slowly coming down for some of the main keywords too
Algorithm Updates | | Modi0 -
Personalization for non logged in users
My question is, how does google personalize the search results for non logged in users and incoqnito searches? I already know about the location personalization whether you're logged in or not and auto complete. But, is personalization still helping in the other cases. Does for example Google keep track of your I.P and then match suggestions that way? Additionally, any other resources would be great.
Algorithm Updates | | PeterRota0 -
How to show your ratings in the Google SERP
I've noticed that some organic search results are showing ratings just above the meta tag. How are these sites doing this? Example: If you search "cash advance", there is a result between #4 and #6 in the organic results. The site is "goldcashadvance.com". It's showing a 5-star rating in the result.
Algorithm Updates | | sparagi0 -
Rich Snippets stopped showing up in SERPS
Up until a few weeks ago my testimonial review ratings (5 star rating system) were showing up in search results but they no longer do. Went to the google rich snippet testing tool and they still do there just not on the real search results. Any thoughts on why? Perhaps an algorithm change?
Algorithm Updates | | casper4340 -
What do you think Google analyzes for SERP ranking?
I've been doing some research trying to figure out how the Google algorithm works. The one thing that is constant is that nothing is constant. This makes me believe that Google takes a variable that all sites have and divides it by that number. One example would be taking the load time in MS and dividing it by the total number or points the website scored. This would give all of the websites a random appearance since there that variable would throw off all the other constants. I'm going to continue doing research but I was wondering what you guys think matters in the Google Algorithm. -Shane
Algorithm Updates | | Seoperior0