Is there a reason to set a crawl-delay in the robots.txt?
-
I've recently encountered a site that has set a crawl-delay command set in their robots.txt file. I've never seen a need for this to be set since you can set that in Google Webmaster Tools for Googlebot. They have this command set for all crawlers, which seems odd to me. What are some reasons that someone would want to set it like that? I can't find any good information on it when researching.
-
Google does not support the crawl delay command directly, but you can lower your crawl priority inside Google Webmaster Central.
So you are right using it the way you are. If you have it in the robots.txt, it does not really do anything and it will show in the webmaster console as well that GWT does not support it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Expired domain 404 crawl error
I recently purchased a Expired domain from auction and after I started my new site on it, I am noticing 500+ "not found" errors in Google Webmaster Tools, which are generating from the previous owner's contents.Should I use a redirection plugin to redirect those non-exist posts to any new post(s) of my site? or I should use a 301 redirect? or I should leave them just as it is without taking further action? Please advise.
Technical SEO | | Taswirh1 -
Robots.txt and Multiple Sitemaps
Hello, I have a hopefully simple question but I wanted to ask to get a "second opinion" on what to do in this situation. I am working on a clients robots.txt and we have multiple sitemaps. Using yoast I have my sitemap_index.xml and I also have a sitemap-image.xml I do put them in google and bing by hand but wanted to have it added into the robots.txt for insurance. So my question is, when having multiple sitemaps called out on a robots.txt file does it matter if one is before the other? From my reading it looks like you can have multiple sitemaps called out, but I wasn't sure the best practice when writing it up in the file. Example: User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Sitemap: http://sitename.com/sitemap_index.xml Sitemap: http://sitename.com/sitemap-image.xml Thanks a ton for the feedback, I really appreciate it! :) J
Technical SEO | | allstatetransmission0 -
First Crawl Report
Just joined SEOMoz today and am slightly overwhelmed, but excited about learning loads from it. I've just received my Crawl Report and there is a
Technical SEO | | iainmoran
404 : UserPreemptionError:
http://www.iainmoran.com/comments/feed/ This is a WordPress site and I've no idea what the best course of action to take. I've done some searching on Google and a couple of sites suggest removing that url from within the robots.txt file. I'm using the Yoast Plugin which apparently creates a robots.txt file, but I can't see any way to edit it. Is there another solution for resolving the 404 error? Many thanks, Iain.0 -
Possible reasons for being outranked?
Suppose you are being outranked for a keyword phrase by a competitor but you have a significantly higher page authority and domain authority. 9 out of 10 people would consider the content better on our page. We have more links pointing to our page from highly authoritative sites than they do. What other reasons could explain why we are being outranked?
Technical SEO | | ProjectLabs0 -
Setting preferred domain as www or none www
Way back before panda I used to rank for certain keywords pretty well. Of course like many others after panda I lost some of those rankings. I have been getting better since then so its not that bad. I was poking around in Google Webmaster Tools and I noticed something which I need some clarification in. History my site freescrabbledictionary.com used to be indexed as a none www. Then some time ago I can't remember when I set it to www. Tonight I was looking through my webmaster tools and I noticed something that did not make sense to me. In my content keywords section for the none www my list is as follows Content Keywords <form action="https://www.google.com/webmasters/tools/keywords-list?hl=en&siteUrl=http://freescrabbledictionary.com/" method="GET"> Keyword Significance 1. scrabble 2. words (2 variants) 3. dictionary 4. cheat 5. finder 6. friends 7. maker (2 variants) 8. noun 9. letter (2 variants) 10. hasbro 11. mattel 12. spear 13. found (2 variants) 14. sowpods 15. freescrabbledictionary 16. builder 17. affiliated 18. search 19. solver 20. lists </form> Then I looked at my www lists and its Content Keywords <form action="https://www.google.com/webmasters/tools/keywords-list?hl=en&siteUrl=http://www.freescrabbledictionary.com/" method="GET"> Keyword Significance 1. words (3 variants) 2. scrabble (2 variants) 3. letter (4 variants) 4. points 5. cheat (3 variants) 6. friends (2 variants) 7. finder (2 variants) 8. anagram (2 variants) 9. dictionary 10. tool (2 variants) 11. hasbro 12. mattel 13. spear 14. game (4 variants) 15. mobile 16. affiliated (3 variants) 17. berkshire 18. canada 19. calculations (5 variants) 20. coming (4 variants) </form> My none www version has the order (especially the first 5 keywords) that I want, my www version is no were near it. If I change back to the none www version could I possible see an change in rank? or can it effect it if I change it? I am starting to think I shot myself in the foot when I switched...
Technical SEO | | cbielich0 -
Homepage outranked by sub pages - reason for concern?
Hey All, trying to figure out how concerned I should be about this. So here is the scoop, would appreciate your thoughts. We have several eCommerce websites that have been affected by Panda, do to content from manufacturers and lack of original content. We have been working hard to write our own descriptions and are seeing an increase in traffic again. We have also been writing blogs since February and are getting a lot of visits to them. Here is the problem, our blog pages are now outranking our homepage when you type in site:domain-name Is this a problem? our home page does not show up until you are 3 pages in. However when you type in just our domain name in google as a search it does show up in position one with sitelinks under it. This is happening across both of our sites. Is this a cause for concern or just natural due to our blogs being more popular than our homepage. Thanks! Josh
Technical SEO | | prima-2535090 -
Severe rank drop due to overwritten robots.txt
Hi, Last week we made a change to drupal core for an update to our website. We accidentally overwrote our good robots.txt that blocked hundreds of pages with the default drupal robots.txt. Several hours after that happened (and we didn't catch the mistake) our rankings dropped from mostly first, second place in Google organic to bottom and mid first page. Basically I believe we flooded the index with very low quality pages at once and threw a red flag and we got de-ranked. We have since fixed the robots.txt and have been re-crawled but have not seen a return in rank. Would this be a safe assumption of what happened? I haven't seen any other sites getting hit in the retail vertical yet in regards to any Panda 2.3 type of update. Will we see a return in our results anytime soon? Thanks, Justin
Technical SEO | | BrettKrasnove0 -
Is blocking RSS Feeds with robots.txt necessary?
Is it necessary to block an rss feed with robots.txt? It seems they are automatically not indexed (http://googlewebmastercentral.blogspot.com/2007/12/taking-feeds-out-of-our-web-search.html) And, google says here that it's important not to block RSS feeds (http://googlewebmastercentral.blogspot.com/2009/10/using-rssatom-feeds-to-discover-new.html) I'm just checking!
Technical SEO | | nicole.healthline0
