How to Set Custom Crawl Rate in Google Webmaster Tools?
-
This is really silly question to set custom crawl rate in Google webmaster tools. Any one can find out that section under setting tab. But, I have confusion to decide number for request per second and second between requests text field.
I want to set custom crawl rate for my eCommerce website.
I checked my Google webmaster tools and find out as attachment. So, Can I use this facility to improve my crawling?
-
Hi Daniel,
I was thinking about it due to very less number of indexing. Thanks for your suggestion.
-
It's better to just leave it on default - custom crawl rates are only needed if you're experiencing traffic problems from the server(as stated in your screenshot).
Yyou will also use more bandwidth on your own server by the google bot coming more often so this should also be something you think about. Also worth mentioning is that after 90 days it will go back to default settings.
http://googlewebmastercentral.blogspot.com/2006/10/learn-more-about-googlebots-crawl-of.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Interest in optimise Google Crawl
Hello, I have an ecommerce site with all pages crawled and indexed by Google. But I have some pages with multiple urls like : www.sitename.com/product-name.html and www.sitename.com/category/product-name.html There is a canonical on all these pages linking to the simplest url (so Google index only one page). So the multiple pages are not indexed, but Google still comes crawling them. My question is : Did I have any interest in avoiding Google to crawl these pages or not ? My point is that Google crawl around 1500 pages a day on my site, but there are only 800 real pages and they are all indexed on Google. There is no particular issue, so is it interesting to make it change ? Thanks
Intermediate & Advanced SEO | | onibi290 -
Does google ignore ? in url?
Hi Guys, Have a site which ends ?v=6cc98ba2045f for all its URLs. Example: https://domain.com/products/cashmere/robes/?v=6cc98ba2045f Just wondering does Google ignore what is after the ?. Also any ideas what that is? Cheers.
Intermediate & Advanced SEO | | CarolynSC0 -
Google News error in Google Search Console
My google search console states some errors as below: 1. Article fragmented Some of the urls in this error are the category urls. How to make google bot understand it is a category not an article? 2. Article too short In fact the article is quite long. I do not know why this is happen... 3. No sentence found In fact, there are a lot of sentences Please help!
Intermediate & Advanced SEO | | binhlai0 -
No images in Google index
No images are indexed on this site (client of ours): http://www.rubbermagazijn.nl/. We've tried everything (descriptive alt texts, image sitemaps, fetch&render, check robots) but a site:www.rubbermagazijn.nl shows 0 image results and the sitemap report in Search Console shows 0 images indexed. We're not sure how to proceed from here. Is there anyone with an idea what the problem could be?
Intermediate & Advanced SEO | | Adriaan.Multiply0 -
We 410'ed URLs to decrease URLs submitted and increase crawl rate, but dynamically generated sub URLs from pagination are showing as 404s. Should we 410 these sub URLs?
Hi everyone! We recently 410'ed some URLs to decrease the URLs submitted and hopefully increase our crawl rate. We had some dynamically generated sub-URLs for pagination that are shown as 404s in google. These sub-URLs were canonical to the main URLs and not included in our sitemap. Ex: We assumed that if we 410'ed example.com/url, then the dynamically generated example.com/url/page1 would also 410, but instead it 404’ed. Does it make sense to go through and 410 these dynamically generated sub-URLs or is it not worth it? Thanks in advice for your help! Jeff
Intermediate & Advanced SEO | | jeffchen0 -
Website No Longer Ranking In Google:
My website was on first page google couple of months ago, now nothing. Shows up in Bing page one. Some queries/pages still showing OK, but some not at all. Example "residential elevators illinois" found nowhere. http://www.accesselevator.net is the website. Have found 900 poor quality links and used disavow tool. Any further suggestions? Their Page Rank also went from a 3 to a 2. Implemented nofollow on all outgoing links. Need advice.
Intermediate & Advanced SEO | | trailblazerzz90 -
Adding video snippet caused drop of star rating snippet in google SERP. Any solution to show both?
We are using on our product pages review schema markup which showed up well in google SERP. Now we added schema video markup as well and google shows in SERP now only the video snippet, but not the review snippet anymore. Any idea whether there may be a solution to show both video and review snippets?
Intermediate & Advanced SEO | | lcourse0 -
How do you find synonyms for a word in Google?
Hi, How would you go about finding a connection between words in the eyes of Google? So for example if I enter ~shoes -shoes I get a few things turn up in bold. Boots is one which makes sense and I would assume Google makes a connection between the words so using both in content would help semantically. BUT the other word it bolds is Store - this is where I get lost why would it make this connection unless Google is taking actual query data from users and making a different type of connection between words. How do you find a solid connection between words?
Intermediate & Advanced SEO | | Bondara0