Can you confirm legitimate Google Bot traffic?
-
We use Cloudflare as a firewall. I noticed a significant number of blocks of bot traffic. One of the things they do is try to block bad bot traffic. But it seems they are mistakenly blocking Google Bot traffic. If you use Cloudflare, you may want to look into this as well. Also, can you confirm if the following IPs are for legitimate Google Bots?
66.249.79.88
66.249.79.65
66.249.79.8066.249.79.76
Thanks,
-
Thank you very much for all of your feedback folks. I really appreciate it. Cloudflare has acknowledged that they are aware of the issue and working to correct it. It also affected Bing Bots. I ended up white listing the IP ranges, and our Google crawl has increased as a result.
-
Dammit, my mistake.
Was proabaly still asleep. I've checked with my log files and updated itThanks Alex!
-
10.0.0.0/8 is a private subnet and therefore unroutable, so I don't think you would have seen traffic from that subnet from Googlebot!
-
What Alex has pointed out is what you're needing.
Also, I'd like to remember you that Googlebot crawls from several IP ranges. Not only the 66.xxx.xxx.xx.
EDIT:
Alex pointed me out that I've made a mistake. So I've checked my data in server logs.
In some of the sites I'm working, most of the traffic comes from the range: [66.249.6x.xxx-66.249.7x.xxx]Always, when in doubt, with a reverse DNSlookup you'll find out whether it's an official GoogleBot or not.
Hope it helps.
Best luck.
Gaston -
Following the advice here: https://support.google.com/webmasters/answer/80553?hl=en I can confirm that they are indeed Googlebot.
If you use Fetch as Google in Search Console, are you seeing errors?
Cloudflare has an article here on the subject:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New website - not showing in Google?
This site was launched 3 days ago, bimcosupply.com and I'm trying to get it to show in Google just for a branded search for the moment (Bimco, Bimco Corporation, etc). The old site is still showing in search, bimcoplumbingsupplies.com instead. This site was taken down a while back. I set up a redirect for the domain in cPanel, and also set individual pages to redirect in WordPress on the bimcosupply.com site. I've verified the site in Google Search Console, submitted a sitemap and did URL inspection on each page. Each page is showing as indexed, though now when I search site:bimcosupply.com not all pages are there, and there are two results for the home page, one "https" and one "http." (Before today, all of the pages were showing so not sure what changed). I know this new domain does not have any (or very little) domain authority yet, but I would have thought that the site should display for branded search by now. So I'm concerned that something is wrong with the site, how the redirects are set up, etc. that is preventing it from displaying. Could anyone take a look and help me figure this out please?
Technical SEO | | browncreative0 -
Google is indexing bad URLS
Hi All, The site I am working on is built on Wordpress. The plugin Revolution Slider was downloaded. While no longer utilized, it still remained on the site for some time. This plugin began creating hundreds of URLs containing nothing but code on the page. I noticed these URLs were being indexed by Google. The URLs follow the structure: www.mysite.com/wp-content/uploads/revslider/templates/this-part-changes/ I have done the following to prevent these URLs from being created & indexed: 1. Added a directive in my Htaccess to 404 all of these URLs 2. Blocked /wp-content/uploads/revslider/ in my robots.txt 3. Manually de-inedex each URL using the GSC tool 4. Deleted the plugin However, new URLs still appear in Google's index, despite being blocked by robots.txt and resolving to a 404. Can anyone suggest any next steps? I Thanks!
Technical SEO | | Tom3_150 -
Can anyone confirm they have predictable rating shifts based on the day of the week?
Hey all, The site in question is https://amygalbraith.com The keyword I am trying to rank for (among others) is 'Seattle Wedding Photographer'. I have seen a truly strange pattern emerge. Each Monday in the past two weeks, this site has been ranked in the top 20, usually around 11-14. Come Tuesday, it will drop completely out of the top 50, usually around page 7-8. Does anyone know of anything that might account for that kind of erratic shift in placement? We aren't making any changes to the site: no a/b testing or anything is occurring at the moment. I'm baffled. It could be a fluke. You can bet I will be trying to get screenshots this coming Monday.
Technical SEO | | amygalbraith0 -
Where can I ... ?
[Spammy content removed by moderator.] Abdullah-Al-Asif-1.jpg?fit=177%2C177&ssl=1 tEjk20Fs_400x400.jpg
Technical SEO | | Abdullah-Al-Asif0 -
Why is Google not indexing my site?
I'm a bit confused as to why my site just isn't indexing on Google. Even if I type in my brand name, my social channels rank and there's no evidence of my website. I've followed all of the advice I've read and gone into webmaster tools and got the Wordpress yoast plug-in but nothing seems to be making a difference!One thing I've noticed, in Google Webmaster Tools it says "Couldn’t communicate with the DNS server." in site errors. I've called GoDaddy and they said that everything is fine. A bit frustrating. Trying to work out what my next steps should be but feeling a bit lost to be honest! Any help GREATLY appreciated!
Technical SEO | | j1066s0 -
Google not indexing my website
Hi guys, We have this website http://www.m-health-expo.nl/ but it is not indexed by google. In webmaster tools google says that it can not fetch the site due to the robots.txt but i do not see any faults in it. http://www.m-health-expo.nl/robots.txt Do you see something strange, it really bothers me.
Technical SEO | | RuudHeijnen0 -
Not ranking well in Google
Hi, I am new to Seomoz,I have some little doubts regarding <title>tag.</p> <p>Can i target 3 words in the title tag. Currently i am on top for one keyword, and i cant get the rest two in top positions. Here is my website, can anyone review my site please.</p> <p>xxx(dot)ridpiles(dot)com with keyword hemorrhoids treatment</p> <p>I have good amount of backlinks, but still something i am missing. I have 100% unique content.</p> <p> </p> <p>Regards</p></title>
Technical SEO | | Dexter22387874870 -
Frustration With Google Places
I have been trying to solve this problem with Google Places for quite some time now and just can't figure out where to go from here. I've tried several sent messages explaining the problem and even received several phone calls from Google Places trying to correct the issue with no luck. I have even tried totally deleting the listing and started over from scratch and re-verified the address with a mailed postcard. My site: http://www.captainrichsmith.com has a Google Places account set up and verified http://maps.google.com/maps/place?hl=en&georestrict=input_srcid:1c8fa43cf77e0c93&ie=UTF8&t=h&z=14&vpsrc=0 For some reason when you do a Google search for one of my keywords Miami Fishing Charters On the listings normally under the letter "E" on the Map another website has a placemark at my location Miami Fishing Charters Directory
Technical SEO | | captainrichsmith
www.fishing-charters-miami.com/ - Cached Fishing Charters Miami is a quality directory of the best fishing boats in the Miami area. The top Miami fishing charters are listed on this website.
2550 South Bayshore Drive, Miami
(786) 263-9231
captainrichsmith.com (7) When you view this Google places listing further. I see it is using my images, videos, placemark on map but NOT the address, phone number, or reviews. Any help on this issue would greatly be appreciated0