Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
-
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.
To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.
Context that might be helpful:
- I did check technical settings, and the sites are crawlable.
- The sites do appear in search but seem to be losing organic search traffic.
Thanks for any help you can provide!
-
Hey David,
I thought I'd jump in here, as it's our tool
We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -
https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots
You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/
Cheers.
Dan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What tools and metrics do you use to show a topic's search interest over time?
I have a foundation repair client that is down in leads for the structural repair portion of their business. They have not lost any major rankings, but leads are down compared to last year. They asked if people are searching for this type of work less this year compared to last. I checked Google Trends and Keyword Planner data but found very different results. Is either of these tools accurate, or is there a better tool to use?
Algorithm Updates | | DigitalDivision1 -
Not showing in Google map listing. Why?
We have a client who's law firm is the highest google reviewed, on page two or three of St. Louis personal injury lawyer, but does not show in the map listing. Any ideas why this would happen or how to ensure they are viewable in the map listing?
Algorithm Updates | | David-Kley0 -
Product shows as out of stock?
Hi, A clients product is showing as out of stock on Google search when I type in the brand/product name. When it is in fact in stock. They have fetched and also fetched and rendered but no joy. Yesterday it went back into stock, then today back out of stock again, but hasn't been out of stock at all on the website, so not sure why it is doing it. When I do the fetch it shows up as in stock at £29.99 on the search it shows as out of stock at £39.99 Platform is EKM Powershop. I contacted them first of all and they insisted the problem was nothing to do with them. Any ideas?
Algorithm Updates | | YNWA0 -
Why isn't our structured markup showing in search results
Hi All, We installed Schema.org structured markup on our pages nearly 1.5 months ago at this point and we have yet to see the markup show in the search results. It also checks out in Webmaster tools and Google's structured markup language testing tool. So, I'm just confused why it's not even showing up site a "site" search in Google either. Here's an example of two such pages on our site: http://www.learningtree.com/htfu/usdc01/washington/java-perl-and-python-programming-training and http://www.learningtree.com/htfu/usat40/alpharetta/it-and-management-training Any advice is greatly appreciated! Thank you 🙂
Algorithm Updates | | CSawatzky0 -
Why is Google.co.uk showing so many US retailers on search queries?
Over the last couple of years Google seems to be showing more and more irrelevant search results in the field I am based in (dietary supplements) with searches for products coming back with results which highlight US vendors. Given I am in google.co.uk and actually logged in as well, why is Google so far off with their results lately?
Algorithm Updates | | predatornutrition0 -
How to show your ratings in the Google SERP
I've noticed that some organic search results are showing ratings just above the meta tag. How are these sites doing this? Example: If you search "cash advance", there is a result between #4 and #6 in the organic results. The site is "goldcashadvance.com". It's showing a 5-star rating in the result.
Algorithm Updates | | sparagi0 -
Meta Title Not Showing up in Google
Hello Friends, I have a website, www.bollywoodshaadis.com. On 1st may we changed our servers and revamped our website as per SEO updated guidelines. For some strange reason Google is not showing site Meta Title when you search the website on Google. All it shows is the domain name in the meta title. However, when you search info:www.bollywoodshaadis.com it shows the right Meta tags. Any reason for this happening? I have never seen this before. Thank you in advance.
Algorithm Updates | | SEOcandy0 -
Does google index non-public pages ie. members logged in page
hi, I was trying to locate resources on the topics regarding how much the google bot indexes in order to qualify a 'good' site on their engine. For example, our site has many pages that are associated with logged in users and not available to the public until they acquire a login username and password. Although those pages show up in google analytics, they should not be made public in the google index which is what happens. In light of Google trying to qualify a site according to how 'engaged' a user is on the site, I would feel that the activities on those member pages are very important. Can anyone offer suggestions on how Google treats those pages since we are planning to do further SEO optimization of those pages. Thanks
Algorithm Updates | | jumpdates0