How to tell how often Google crawls someone else's site
-
How can I tell how often Google crawls someone else's site?
-
To get a rough idea , check the cache date in SERPS for a couple of days ad see how often that changes.
-
Dear Friend,
You can do this by browsing on your WEB log files and looking for the string "http://www.google.com/bot.html"
How often your site is crawled by google depends on many factors ie.:
1. How often you add or modify content.
2. How many links are pointing to your site.
3. and more.
In my experience this crawler freq. is a good signal of active and quality site.
Hope it help
Claudio
-
You can't -- but the better question is why does it matter/is it important to you?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New SEO manager needs help! Currently only about 15% of our live sitemap (~4 million url e-commerce site) is actually indexed in Google. What are best practices sitemaps for big sites with a lot of changing content?
In Google Search console 4,218,017 URLs submitted 402,035 URLs indexed what is the best way to troubleshoot? What is best guidance for sitemap indexation of large sites with a lot of changing content? view?usp=sharing
Technical SEO | | Hamish_TM1 -
How do you 'close down' a website?
Hello all, If a company acquires a smaller company and 'absorbs' its products and services into its own website, what is the protocol with closing down the smaller company's site? So far we added our branding to the site alerting their visitors to the imminent takeover, and 301 redirected certain pages - soon we'll be redirecting all the pages to their counterparts on the main website. Once that's done, should we noindex the old site? Anything else? Thanks, Caro
Technical SEO | | Caro-O0 -
Why Can't Googlebot Fetch Its Own Map on Our Site?
I created a custom map using google maps creator and I embedded it on our site. However, when I ran the fetch and render through Search Console, it said it was blocked by our robots.txt file. I read in the Search Console Help section that: 'For resources blocked by robots.txt files that you don't own, reach out to the resource site owners and ask them to unblock those resources to Googlebot." I did not setup our robtos.txt file. However, I can't imagine it would be setup to block google from crawling a map. i will look into that, but before I go messing with it (since I'm not familiar with it) does google automatically block their maps from their own googlebot? Has anyone encountered this before? Here is what the robot.txt file says in Search Console: User-agent: * Allow: /maps/api/js? Allow: /maps/api/js/DirectionsService.Route Allow: /maps/api/js/DistanceMatrixService.GetDistanceMatrix Allow: /maps/api/js/ElevationService.GetElevationForLine Allow: /maps/api/js/GeocodeService.Search Allow: /maps/api/js/KmlOverlayService.GetFeature Allow: /maps/api/js/KmlOverlayService.GetOverlays Allow: /maps/api/js/LayersService.GetFeature Disallow: / Any assistance would be greatly appreciated. Thanks, Ruben
Technical SEO | | KempRugeLawGroup1 -
What can i do to get google to visit my site more often
Hi, i am having serious problems since i upgraded my website from joomla 1.5 to 3.0 We have dropped down the rankings from page one for the word lifestyle magazine, and we have dropped down in rankings for other very important words including gastric band hypnotherapy and i am starting to regret having the site upgraded. i am finding the google is taking its time visiting my site, i know this for two reasons, one i have checked the cache and it is showing the 2nd july and i have checked articles that we have written and they are still not showing. example if i put this article name in word for word it does not come up, Carnival Divert Ships In The Caribbean Due To bad Weather this was an article that was done yesterday. in the old days before the upgrade that would have been in google now. these problems are costing us the loss of a great deal of traffic, we are losing around 70% of our traffic since the upgrade and would be grateful if people could give me advice on how to turn things around. we add articles all the time. each day we add a number of articles, i was considering changing the front page in the middle and having a few paragraphs of the latest story to get google to visit more often. i know this would look messy but i am running out of ideas. any help would be great
Technical SEO | | ClaireH-1848860 -
Should I worry about these 404's?
Just wondering what the thought was on this. We have a site that lets people generate user profiles and once they delete the profile the page then 404's. I was told there is nothing we can do about those from our developers, but I was wondering if I should worry about these...I don't think they will affect any of our rankings, but you never know so I thought I would ask. Thanks
Technical SEO | | KateGMaker1 -
How can I tell Google, that a page has not changed?
Hello, we have a website with many thousands of pages. Some of them change frequently, some never. Our problem is, that googlebot is generating way too much traffic. Half of our page views are generated by googlebot. We would like to tell googlebot, to stop crawling pages that never change. This one for instance: http://www.prinz.de/party/partybilder/bilder-party-pics,412598,9545978-1,VnPartypics.html As you can see, there is almost no content on the page and the picture will never change.So I am wondering, if it makes sense to tell google that there is no need to come back. The following header fields might be relevant. Currently our webserver answers with the following headers: Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0, public
Technical SEO | | bimp
Pragma: no-cache
Expires: Thu, 19 Nov 1981 08:52:00 GMT Does Google honor these fields? Should we remove no-cache, must-revalidate, pragma: no-cache and set expires e.g. to 30 days in the future? I also read, that a webpage that has not changed, should answer with 304 instead of 200. Does it make sense to implement that? Unfortunatly that would be quite hard for us. Maybe Google would also spend more time then on pages that actually changed, instead of wasting it on unchanged pages. Do you have any other suggestions, how we can reduce the traffic of google bot on unrelevant pages? Thanks for your help Cord0 -
It's imposible to keep track of rankings?
Hello, here something interesting I'm Using Rank Tracker from SEOMOZ And from the link-assistant's Rank Tracker as Well... I need to track Google.com and Google.co.ve (venezuela) so I did... i got my keyword an here are my results. 1 Keyword A at google.com (united states) Rank Tracker SEOMOZ = pos 6 Rank Tracker OTHER = pos 6 Manual Query on google.com = 9 (I used the exact url seomoz tells me its using) 2 Keyword A at Google.co.ve Rank Tracker SEOMOZ = pos 8 Rank Tracker OTHER = pos 7 Manual query on google.co.ve = pos 8 So.... Why it's that?, so far I think that google.com for me down here (it actually says "Español") it's a different index? for latinamerica? only spanish pages? maybe it's because there's a couple of minutes between looking with one tool and the other... any help, would be great... Dan
Technical SEO | | daniel.alvarez0 -
Google crawl index issue with our website...
Hey there. We've run into a mystifying issue with Google's crawl index of one of our sites. When we do a "site:www.burlingtonmortgage.biz" search in Google, we're seeing lots of 404 Errors on pages that don't exist on our site or seemingly on the remote server. In the search results, Google is showing nonsensical folders off the root domain and then the actual page is within that non-existent folder. An example: Google shows this in its index of the site (as a 404 Error page): www.burlingtonmortgage.biz/MQnjO/idaho-mortgage-rates.asp The actual page on the site is: www.burlingtonmortgage.biz/idaho-mortgage-rates.asp Google is showing the folder MQnjO that doesn't exist anywhere on the remote. Other pages they are showing have different folder names that are just as wacky. We called our hosting company who said the problem isn't coming from them... Has anyone had something like this happen to them? Thanks so much for your insight!
Technical SEO | | ILM_Marketing
Megan0