Thanks for your response.  I get where you're going with that.   (Ecomm store gone bad.)    It's not actually an Ecomm FWIW.  And I do restrict parameters - the list is about a page and a half long.  It's a legitimately large site.

You're correct - I don't want Google to crawl the full 500M.   But I do want them to crawl 100M.  At the current crawl rate we limit them to, it's going to take Google more than 3 months to get to each page a single time.  I'd actually like to let them crawl 3M pages a day.   Is that an insane amount of Googlebot bandwidth?   Does anyone else have a similar situation?

One of our sites is very large - over 500M pages.   Google has indexed 1/8th of the site - and they tend to crawl between 800k and 1M pages per day.

A few times a year, Google will significantly increase their crawl rate - overnight hitting 2M pages per day or more.  This creates big problems for us, because at 1M pages per day Google is consuming 70% of our API capacity, and the API overall is at 90% capacity.   At 2M pages per day, 20% of our page requests are 500 errors.

I've lobbied for an investment / overhaul of the API configuration to allow for more Google  bandwidth without compromising user experience.   My tech team counters that it's a wasted investment - as Google will crawl to our capacity whatever that capacity is.

Questions to Enterprise SEOs:

*Is there any validity to the tech team's claim?  I thought Google's crawl rate was based on a combination of PageRank and the frequency of page updates.   This indicates there is some upper limit - which we perhaps haven't reached - but which would stabilize once reached.

*We've asked Google to rate-limit our crawl rate in the past.   Is that harmful?  I've always looked at a robust crawl rate as a good problem to have.

• Is 1.5M Googlebot API calls a day desirable, or something any reasonable Enterprise SEO would seek to throttle back?

*What about setting a longer refresh rate in the sitemaps?   Would that reduce the daily crawl demand?  We could set increase it to a month, but at 500M pages Google could still have a ball at the 2M pages/day rate.

Thanks

So it seems like we've gone full circle.

The initial question was, "How can I tell Google (and other crawlers) to temporarily (a day or two) ignore my site?  We're hoping to buy ourselves a small bit of time to verify redirects and live functionality before allowing Google to crawl and index the new architecture."

Sounds like the answer is, 'that's not possible'.

Thank you, Ruth!

Can I ask a clarifying question?

If I put a noindex/nofollow on the new urls, wouldn't the result be the same as if I put noindex/nofollow on the indexed urls?  There is only one instance of each page - and all of the millions of  indexed URLs will be redirecting to new urls.

Here is my assumption:   if I put noindex/nofollow on the new urls - a search bot will crawl the old url, follow the redirect to the new url, detect the noindex/nofollow, and then drop the old, indexed url from their index.  Is that the wrong assumption?

Thanks for the response, Mark.

It sounds as if you tried this on a few new pages.

I'm talking about millions of existing pages.

Would you robots.txt noindex your entire website?   Seems like you'd run a huge risk of being dumped from the index entirely.

We'll soon be launching a redesign, on a new platform, migrating millions of pages to new URLs.

How can I tell Google (and other crawlers) to temporarily (a day or two) ignore my site?  We're hoping to buy ourselves a small bit of time to verify redirects and live functionality before allowing Google to crawl and index the new architecture.

GWT's recommendation is to 503 all pages - including robots.txt, but that also makes the site invisible to real site visitors, resulting in significant business loss.  Bad answer.

I've heard some recommendations to disallow all user agents in robots.txt.   Any answer that puts the millions of pages we already have indexed at risk is also a bad answer.

Thanks

We're a publisher, which means that as an industry our normal render times are always at the top of the chart.  Ads are notoriously slow to load, and that's how we earn our keep.  These results are bad, though, even for publishing.

We're serving millions of uniques a month, on a bank of dedicated servers hosted off site, load balanced, etc.

We’ve been noticing a huge uptick in Google’s spidering lately, and along with it a notable worsening of render times.

Yesterday, for example, Google spidered our site at a rate of 30:1 (google spider vs. organic traffic.)   So in other words, for every organic page request, Google hits the site 30 times.

Our render times have lengthened to an avg. of 2 seconds (and up to 2.5 seconds). Before this renewed interest Google has taken in us we were seeing closer to one second average render times, and often half of that.

A year ago, the ratio of Spider to Organic was between 6:1 and 10:1.

Is requesting a crawl-delay from Googlebot a viable option?

Our goal would be only to reduce Googlebot traffic, and hopefully improve render times and organic traffic.

Thanks,

Trisha