Suggested crawl rate in google webmaster tools?
-
hey moz peeps, got a general question:
what is the suggested custom crawl rate in google webmaster tools? or is it better to "Let Google determine my crawl rate (recommended)"
If you guys have any good suggestions on this and site why that would be very helpful, thanks guys!
-
Hi David,
According to Google's webmasters help page, it states that the crawl rate option is just about how fast it crawls your site and not how often it does. Check out the link for more information about it.
Also, I wouldn't stress to much on the settings because the search engines are going to do what they do, just make sure to create good content and get it out there, the search engines will see this and depending on how well it is they we come more often on there own.
Hope this helps!
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=48620
-
thanks for that alan, yea id like to see what other have to say about this as well, im all open ears!
-
I believe it is much better to allow the search engines to handle it themselves. Check your logs to see what they are doing, so you know if they are hammering you server. If they are, you can slow them down using the robots.txt file On a large site, google will often such out a few hundred thousand pages per day. Yahoo will do more, but their crawler isn't always friendly or smart. Sometimes different Slurp crawlers will request the same pages. Bing emailed me six months ago and asked that we remove the 2second delay I added. I did that because they were hammering the server, but only sent a few thousand readers per day. Google doesn't usually hammer the server, but when 5 or 6 are hitting you at the same time, it can affect you site response time. - and then some of them want to penalize you for having a poor experience for visitors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Manual Webspam Error. Same Penalty on all sites on Webmaster Tools account.
My URL is: www.ebuzznet.comToday when i checked webmaster tools under manual spam section. I got Manual spam and reason was Thin content with little or no added value. Then I checked other sites from same webmaster tools account, there are 11 sites, all of them received same manual action. I never received any mail, no notification on site messages section regarding this manual action. I just need confirmation whether it is something to do with any error in webmaster tools or all of the sites really received manual spam actions.Most of the article on sites are above 500 words, quality content (no spun or copied).Looking for suggestions, answers
Technical SEO | | ndroidgalaxy0 -
Google dance tools - do they still give an indication of rankings to come?
I can't remember the tool I used in the past but it searched a number of data centres which gave a pretty good indication of what your new rankings would be once full updated, is this still the case today? If so, can anyone recommend any tools?
Technical SEO | | designguy760 -
Is it OK for a sitemap to appear as a "Top URL" in Google Webmaster?
I'm using Google Webmaster (alongside other tools) to understand how Google is indexing my site. One of the tools is "Content Keywords", where it lists keywords that Google sees as significant for your site. The keywords shown are generally fine, but when I click on an individual word, I am often seeing our sitemap as one of the "Top URLs" that the keyword is found on (our sitemap is at system/sitemap1.xml.gz) - is this OK? Obviously I don't want to add the sitemap URL to robots.txt, but I also want to ensure that 'real' user-focused pages (e.g. our homepage) appear higher in the "Top URLs" list for the keywords, as I'm assuming this is an indicator of how the site is performing in search. Any help appreciated!
Technical SEO | | anilababla0 -
Why Google not picking My META Description? Google itself populate the description.. How to control this Search Snippets??
Why Google not picking My META Description? Google itself populate the description.. How to control this Search Snippets??
Technical SEO | | greyniumseo0 -
Broken LInks Tool?
Hi I am trying to find broken links on my site, Is there a tool for this? Free or Paid tool is fine thanks.
Technical SEO | | daracreative0 -
Crawl diagnostic summary
In my crawl diagnostic summary its showing an error with duplicate page title and duplicate page content...why its been shown and how it can be rectified? I have pne page web site so i was unable to give options for sub domain name is it because of tht?I hope this error wont hamper my SEO process.
Technical SEO | | strasshgoa0 -
Google not visiting my site
Hi my site www.in2town.co.uk which is a lifestyle magazine has gone under a major refit. I am still working on it but it should be ready by the end of this week or sooner but one problem i have is, google is not visiting the site. I took a huge gamble to redo the site, even though before the refit i was getting a few thousand visitors a day, i wanted to make the site better as i was getting google webmaster errors. But now it seems google is not visiting the site. for example i am using sh404sef and i have put friendly url in the site and on the home page it has its name and meta tag but when you look at google it is not giving the site a name. Also it has not visited the site since october 13th Can anyone advise how to encourage google to visit the site please.
Technical SEO | | ClaireH-1848860 -
About Google Spider
Hello, people! I have some questions regarding on Google spider. Many people are saying that "Google spiders only have US IP address." Is this really true? But I also saw video from Google's offical blog and it said "Google spider come from all around the world." At this point I am really confused. Q1) I researched and it seems like Google spiders have only US IP address. THen what does exactly mean by "Google spider come from all around the world."? Q2) If Google spider have only US IP address, what happen to site which use IP delivery? Is this means that Google spider always redirect to us site since they only have US IP? Can anyone help me to understand?? One more questions! When Google analyzing for cloaking issue, do you think Google analyze when spider crawls the site or after they crawled the site?
Technical SEO | | Artience0