Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
-
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.
To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.
Context that might be helpful:
- I did check technical settings, and the sites are crawlable.
- The sites do appear in search but seem to be losing organic search traffic.
Thanks for any help you can provide!
-
Hey David,
I thought I'd jump in here, as it's our tool
We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -
https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots
You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/
Cheers.
Dan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Not showing in Google map listing. Why?
We have a client who's law firm is the highest google reviewed, on page two or three of St. Louis personal injury lawyer, but does not show in the map listing. Any ideas why this would happen or how to ensure they are viewable in the map listing?
Algorithm Updates | | David-Kley0 -
How do you get a url to show as a tagline in google mobile search?
When searching in google via mobile, I am seeing urls changed to taglines. I have attached pictures that show the url in a web search, but a tag line from the mobile search. Does anyone know how to get a tagline to show in place of a url in a mobile search? Any advice would be appreciated! uLkYWRx.png wljXRI3.png
Algorithm Updates | | David-Kley0 -
Category Containing a Product searched shows up higher in google then the product page itself?
Hello Moz Wizards, We have recently launched a new eCommerce website www.memoky.com and think we did a pretty good job with the markup structure for feeding the hungry google bot all information available about a the products. However google doesn't like us very much : ( It seems every time you google a product that we carry; the category pages that contain that product will show up, but the product page itself does not. Below are two examples, however this seems to be site-wide which makes me feel like there is an underlying issue that we are missing. Examples
Algorithm Updates | | Memoky
when searched for "Eduardo floor lamp - matt black/matt yellow shade"
Shows ups - http://www.memoky.com/lighting/floor-lamps.html
Does not - http://www.memoky.com/eduardo-floor-lamp-matt-black-matt-yellow-shade.html when searched for "Derrick arm chair - white leather/ walnut"
Shows ups - http://www.memoky.com/living/lounge-chairs.html_
Does not - http://www.memoky.com/derrick-arm-chair-white-leather.html_ that is the pattern for almost all the products on this site. Any thoughts on why this could be the case?0 -
Need to be reindexed quickly - SERP is showing a 404
So there was a mistake made where a 404 error was placed in the canonical URL for the pages my company made. We need to have these pages quickly reindexed. I asked GWT to fetch them and have an updated sitemap but the SERPs are still the same. Any tricks anyone knows that would allow me to get reindexed faster?
Algorithm Updates | | mattdinbrooklyn0 -
Why isn't our structured markup showing in search results
Hi All, We installed Schema.org structured markup on our pages nearly 1.5 months ago at this point and we have yet to see the markup show in the search results. It also checks out in Webmaster tools and Google's structured markup language testing tool. So, I'm just confused why it's not even showing up site a "site" search in Google either. Here's an example of two such pages on our site: http://www.learningtree.com/htfu/usdc01/washington/java-perl-and-python-programming-training and http://www.learningtree.com/htfu/usat40/alpharetta/it-and-management-training Any advice is greatly appreciated! Thank you 🙂
Algorithm Updates | | CSawatzky0 -
Personalization for non logged in users
My question is, how does google personalize the search results for non logged in users and incoqnito searches? I already know about the location personalization whether you're logged in or not and auto complete. But, is personalization still helping in the other cases. Does for example Google keep track of your I.P and then match suggestions that way? Additionally, any other resources would be great.
Algorithm Updates | | PeterRota0 -
Search bots that use referrers?
Can someone point me to a list or just tell me specific search bots that use referrers?
Algorithm Updates | | BostonWright0 -
Anyone have stats on numbers of Google users searching while logged in?
In light of Google's recent "social search update", I am curious to know how many Google users perform searches while logged into their Google account thereby showing "social results".
Algorithm Updates | | Gyi0