Log File Analyzer Only Showing Spoofed Bots and No Verified Bots
-
Question for you guys: After analyzing some crawl data in Search Console in the sitemap section, I noticed that Google consistently isn't indexing about 3/4 of the client sites I work on that all use the same content management system. I began to wonder if maybe Google (and others) have a hard time crawling certain parts of the sites consistently, as finding a pattern here could lead me to investigate whether there's a CMS problem.
To research this, I started using a log file analyzer (Screaming Frog's version) for some of those clients. After loading the files, I noticed that none of the crawl activity logged by the servers is considered verified. I input one month's worth of log files, but when I switch the program to show only verified bots, all data disappears. Is it possible for a site not to have any search engines crawling it for a whole month? Given my experience, that seems unlikely, particularly since we've been submitting crawl requests. I know that doesn't guarantee a crawl, but it seems odd that it's never happening for any search engines across the board.
Context that might be helpful:
- I did check technical settings, and the sites are crawlable.
- The sites do appear in search but seem to be losing organic search traffic.
Thanks for any help you can provide!
-
Hey David,
I thought I'd jump in here, as it's our tool
We have more information on bot verification here, including a troubleshooting section with common issues for genuine events being marked as spoofed -
https://www.screamingfrog.co.uk/log-file-analyser/user-guide/configuration/#verify-bots
You can also reach us via our support here - https://www.screamingfrog.co.uk/log-file-analyser/support/
Cheers.
Dan
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site:www Issue - Homepage of the website is not showing in Google
Hello everyone, When I have manually search site:www.blinds4uk.co.uk in google.co.uk to know about webpages status, home page of the website is not showing in google search engine result pages. Please let me know, what is the reason behind this? because website crawling and indexing is good. Many Thanks.
Algorithm Updates | | Kuldeep-Sharma0 -
Google Webmaster Tools show the error in Manual Action while there is no any error in Structured Data Testing Tool.
It is showing error as below Spammy structured markup Markup on some pages on this site appears to use techniques such as marking up content that is invisible to users, marking up irrelevant or misleading content, and/or other manipulative behavior that violates Google's Spammy Structured Markup guidelines. While I see in Structured Data Testing Tool, it doesn't show any error.
Algorithm Updates | | infinitemlm0 -
Not showing in Google map listing. Why?
We have a client who's law firm is the highest google reviewed, on page two or three of St. Louis personal injury lawyer, but does not show in the map listing. Any ideas why this would happen or how to ensure they are viewable in the map listing?
Algorithm Updates | | David-Kley0 -
Adding non-important folders to disallow in robots.txt file
Hi all, If we have many non-important folders like /category/ in blog.....these will multiply the links. These are strictly for users who access very rarely but not for bots. Can we add such to disallow list in robots to stop link juice passing from them, so internal linking will me minimised to an extent. Can we add any such paths or pages in disallow list? Is this going to work pure technical or any penalty? Thanks, Satish
Algorithm Updates | | vtmoz0 -
Product shows as out of stock?
Hi, A clients product is showing as out of stock on Google search when I type in the brand/product name. When it is in fact in stock. They have fetched and also fetched and rendered but no joy. Yesterday it went back into stock, then today back out of stock again, but hasn't been out of stock at all on the website, so not sure why it is doing it. When I do the fetch it shows up as in stock at £29.99 on the search it shows as out of stock at £39.99 Platform is EKM Powershop. I contacted them first of all and they insisted the problem was nothing to do with them. Any ideas?
Algorithm Updates | | YNWA0 -
Need to be reindexed quickly - SERP is showing a 404
So there was a mistake made where a 404 error was placed in the canonical URL for the pages my company made. We need to have these pages quickly reindexed. I asked GWT to fetch them and have an updated sitemap but the SERPs are still the same. Any tricks anyone knows that would allow me to get reindexed faster?
Algorithm Updates | | mattdinbrooklyn0 -
Same search term shows #1 on Bing but #140 on Google?
Hi, I am using the search term of my website domain i.e. "Series Digital" on both Bing and Google. Bing shows my website as the top most link. But on Google, my website appears on page 14!! Why is this happening when I am using the string within the " "?
Algorithm Updates | | Cloudguru990 -
Ecommerce good/bad? Showing product description on sub/category page?
Hi Mozers, I have a ecommerce furniture website, and I have been wondering for some time if showing the product descriptions on the sub/category page helps the website. If there is more content displayed on the subcategory, it should be more relevant, right? OR does it not matter, as it is duplicate content from the product page. I think showing the product descriptions on non-product pages is hurting my design/flow, but i worry that if I am to hide product content on sub/category pages my traffic will be hurt. Despite my searches I have not found an answer yet. Please take a look at my site and share your thoughts: http://www.ecustomfinishes.com/ Chris 27eVz
Algorithm Updates | | longdenc_gmail.com0