Can you confirm legitimate Google Bot traffic?
-
We use Cloudflare as a firewall. I noticed a significant number of blocks of bot traffic. One of the things they do is try to block bad bot traffic. But it seems they are mistakenly blocking Google Bot traffic. If you use Cloudflare, you may want to look into this as well. Also, can you confirm if the following IPs are for legitimate Google Bots?
66.249.79.88
66.249.79.65
66.249.79.8066.249.79.76
Thanks,
-
Thank you very much for all of your feedback folks. I really appreciate it. Cloudflare has acknowledged that they are aware of the issue and working to correct it. It also affected Bing Bots. I ended up white listing the IP ranges, and our Google crawl has increased as a result.
-
Dammit, my mistake.
Was proabaly still asleep. I've checked with my log files and updated itThanks Alex!
-
10.0.0.0/8 is a private subnet and therefore unroutable, so I don't think you would have seen traffic from that subnet from Googlebot!
-
What Alex has pointed out is what you're needing.
Also, I'd like to remember you that Googlebot crawls from several IP ranges. Not only the 66.xxx.xxx.xx.
EDIT:
Alex pointed me out that I've made a mistake. So I've checked my data in server logs.
In some of the sites I'm working, most of the traffic comes from the range: [66.249.6x.xxx-66.249.7x.xxx]Always, when in doubt, with a reverse DNSlookup you'll find out whether it's an official GoogleBot or not.
Hope it helps.
Best luck.
Gaston -
Following the advice here: https://support.google.com/webmasters/answer/80553?hl=en I can confirm that they are indeed Googlebot.
If you use Fetch as Google in Search Console, are you seeing errors?
Cloudflare has an article here on the subject:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Blocking Google from telemetry requests
At Magnet.me we track the items people are viewing in order to optimize our recommendations. As such we fire POST requests back to our backends every few seconds when enough user initiated actions have happened (think about scrolling for example). In order to eliminate bots from distorting statistics we ignore their values serverside. Based on some internal logging, we see that Googlebot is also performing these POST requests in its javascript crawling. In a 7 day period, that amounts to around 800k POST requests. As we are ignoring that data anyhow, and it is quite a number, we considered reducing this for bots. Though, we had several questions about this:
Technical SEO | | rogier_slag
1. Do these requests count towards crawl budgets?
2. If they do, and we'd want to prevent this from happening: what would be the preferred option? Either preventing the request in the frontend code, or blocking the request using a robots.txt line? The latter question is given by the fact that a in-app block for the request could lead to different behaviour for users and bots, and may be Google could penalize that as cloaking. The latter is slightly less convenient from a development perspective, as all logic is spread throughout the application. I'm aware one should not cloak, or makes pages appear differently to search engine crawlers. However these requests do not change anything in the pages behaviour, and purely send some anonymous data so we can improve future recommendations.0 -
Homepage is deindexed in Google
We recently noticed that our primary page was de-indexed in Google. When looking in google search console there are no manual actions taken. We did add a few new banners to the site but I have no idea why this would have negatively affected that site. I did add a new page called https://enleaf.com/company/testimonials/ that had some duplicate testimonials that were also on the home page but have since removed that. Not sure where to go from here.
Technical SEO | | AChronister0 -
Vanity URLs are being indexed in Google
We are currently using vanity URLs to track offline marketing, the vanity URL is structured as www.clientdomain.com/publication, this URL then is 302 redirected to the actual URL on the website not a custom landing page. The resulting redirected URL looks like: www.clientdomain.com/xyzpage?utm_source=print&utm_medium=print&utm_campaign=printcampaign. We have started to notice that some of the vanity URLs are being indexed in Google search. To prevent this from happening should we be using a 301 redirect instead of a 302 and will the Google index ignore the utm parameters in the URL that is being 301 redirect to? If not, any suggestions on how to handle? Thanks,
Technical SEO | | seogirl221 -
How to rank in Google Places
Normally, I don't have a problem with local SEO (more of a multi-channel sort of online marketing guy) but this one has got me scratching my head. Look at https://www.google.co.uk/search?q=wedding+venues+in+essex Theres two websites there (fennes and quendon park) that both have a much more powerful DA but don't appear in the Google Places (Google + Business or whatever it's labeled as). Why are websites such as Boreham house ranking top in the map listings? Quendon Park has a Google places listing, it's full of content, the NAP all matches up. Its a stronger website. Boreham House isn't any closer to the centroid than Quendon Park Just got me struggling this one
Technical SEO | | jasonwdexter0 -
Google is changing the title
Hi! Lately i have seen that Google changed the page titles for some clients, not all... its about 30% of them. For example, the title looks likes this after the Google change: Company name: SEO and Pay per click management But in on that page it looks like this: SEO and Pay per click management - Company name Does anyone know why?
Technical SEO | | DanielNordahl1 -
Google News Referral Traffic Gone after CMS Upgrade
Hoping someone can help. I've tried the Google forum which is it's usual vacuous hole of no help. We recently upgraded our CMS which now includes automated sitemaps (not sure if this impacts news sitemaps, though). Anyway, after rollout, we're no longer getting any traffic from Google News. Sequence of Events: 1. Dec. 13 launch 2. Google news referral traffic stopped 3. IT tried to 'fix' and now we're getting the elusive 'publication name' message The publication name specified in your Sitemap does not match the publication name we have for your site in our database. Please check that the publication name in your Sitemap exactly matches the publication name displayed for your articles in Google News. 4. Retested sitemap When I retest the news sitemap, it returns 0 errors. When I do a site:mnn.com (our website) on the Google New Homepage, new content is showing up in SERP Yet, still no google news referral traffic. Our news sitemap is www.mnn.com/googlenews.xml Can someone help? We're a news site, and can't afford to lose this traffic. Would love an early Christmas present. Thanks!! Lisa
Technical SEO | | Aggie0 -
Unexplained spikes in Google Analytics
My site has modest traffic (50 unique visitors per day). In the past week, I've seen two unexplained spikes in my Google Analytics. Yesterday, there were 140 unique visitors, and these unique visitors each visited one unique page. This appears to be a bot of some sort. If this is a bot, why does Google Analytics think these are unique visitors? Is there a was for small sites to deal with this? Best,
Technical SEO | | ChristopherGlaeser
Christopher0 -
Should I delete Google Places and start over?
My Google Places page is stagnant. It never changes its rank, doesn't find corresponding reviews on directory sites, disappears altogether after a few days for certain keywords and ranks poorly overall. We have had a horrible time getting our web presence to be uniform...name, address, phone, etc. Our business was previously a different name so all those listings were still active, we have had different doctors over the years...those were associated with the business name in different listings, we used a referral service that sponsored listings using a diffferent phone number, change web url in the past year so some sites indicated the wrong address. HUGE HEADACHE Are there any positives/negatives to deleting the Places page and starting over? Here is the page: http://maps.google.com/maps/place?hl=en&georestrict=input_srcid%3Ab1706925095a8afa Thank you for your help!
Technical SEO | | PMC-3120871