Googlebot CDN Migration: Avoid Blocks and Origin Overload

For a Googlebot CDN migration, you can avoid Googlebot blocks and origin overload by following the three steps Google laid out, in order. On November 11, 2026, Google Search Central explained on its official blog why crawling rises right after you move a site behind a CDN, and how to respond. Allow verified bots in your defenses, fill the cache before the switch, and if the server still cannot cope, return 429 or 503 for a short time only. A CDN (Content Delivery Network) is a service where servers spread around the world serve your site files close to visitors. The origin is your own server that holds the original content.

Why a Googlebot CDN migration increases crawling

Google keeps watching whether your server is healthy and sets an upper limit on the crawl rate. This limit is called the crawl capacity limit (also known as hostload). According to the official post, the limit goes down when responses slow down. When fast, stable responses continue, the limit rises gradually as long as there is crawl demand.

When DNS points to the CDN edge servers, connection and response times drop. Googlebot reads this as a signal that the server can take more requests, so it sends more. The problem is that the new CDN cache is empty. The table below shows the difference between visitors and crawlers that Google described.

Item	Human visitors	Googlebot
Requests target	Concentrated on popular pages	Sweeps all URLs on the site
Cache hits	Popular pages fill quickly and get fast	With so many URLs, requests not in the cache (cache misses) continue for a long time
Origin load	Small	Large until the cache fills

Here is a calculation example (an assumption). On a shop with 100,000 pages where visitors mostly view 500 products, once those 500 are cached, visitor requests almost never reach the origin. Googlebot requests all 100,000, so every URL not in the cache must be fetched from the origin. Google did not publish specific figures such as the size of the increase.

Googlebot CDN migration tip: allow verified bots

When the request graph spikes, you may want to switch on bot protection across the board in the CDN dashboard. The official post points to the problem in the name itself: blanket protection. Blanket protection with no exceptions treats Google crawlers as the same automated traffic, putting them in front of error pages or CAPTCHAs. Crawlers back off when this continues, and pages can drop out of search.

The order is as follows.

1. Check whether bot blocking rules are turned on in your CDN and WAF settings. Google noted that most major CDNs and WAFs offer a switch or managed rule that allows known, verified bots.
2. Turn that option on, and block bots you need to block separately with robots.txt or rules aimed at specific bots.
3. If you must identify Googlebot yourself, use the IP ranges Google publishes. The Google request verification document describes how to run a reverse DNS lookup (the host command) on the IP in your log, check that the name ends in googlebot.com, then look that name up again and compare it with the original IP. It also provides IP range files such as common-crawlers.json.

Setting names differ between CDNs and WAFs, so look at the function rather than the name. What to check is whether your bot defense rules have a verified crawler exception, and whether Googlebot requests show up as blocked in the rule logs.

Warm the cache before the switch

In the official post, Google said that when the cache is empty, every first request goes to the origin. So before you send all traffic to the CDN, it recommended requesting key URLs programmatically to fill the cache first. Google named three types of targets.

Warm-up target	Example
Most visited pages	Home page, popular posts, popular product detail pages
Pages about to get traffic	Promotion or seasonal event pages
URLs Googlebot fetches often	Categories, key URLs listed in the sitemap

Send the requests at a controlled rate. If the warm-up script fires everything at once, it becomes an origin overload itself. To find the URLs Googlebot fetches often, count Googlebot requests per URL in your server logs.

Temporary throttling with 429 and 503: how long is safe

If the origin still cannot cope after warming and allow settings, Google said to return 429 (too many requests) or 503 (service temporarily unavailable) for a short time. According to the document on reducing the crawl rate, when Googlebot gets these responses it slows crawling for the whole host, and it speeds up again once the errors stop.

Response	When to use	Note
429, 503	Temporary throttling to protect the origin	A few hours, one or two days at most. The Retry-After header can tell the crawler when to retry
403	Do not use	This is the wall that appears when a WAF or CAPTCHA blocks crawlers. Google did not recommend it as a throttling method

Duration is the key. If these errors continue on the same URL for several days, that URL can drop out of the Google index, discovery of new pages and pickup of edits slow down, and deleted pages stay in the index longer. It is safest to turn on throttling rules with a release date already set. If you only want to lower the speed without errors, per Google's documentation you can request an appropriate crawl rate through the Googlebot report form in Search Console. Review takes a few days, and it cannot be used to ask for a higher rate.

How to check the migration worked

The checks below are not wording from the official post. They turn the guidance above into a checklist. There are three success criteria. In your server or CDN logs, Googlebot requests end in 200, not in 403 or a challenge page. Origin request counts and response times come down within a few days of the switch. In the Crawl Stats report in Search Console, server errors and average response time do not visibly get worse. A rise in crawl requests is itself the normal reaction Google described.

Frequently asked questions

Does using a CDN raise my SEO rankings?

The official post does not talk about rankings. It only explains that when responses get faster, Googlebot raises its crawl rate.

Do small blogs need cache warming?

If you have few pages and the server has headroom, you probably do not. What to check is whether your bot defense settings are blocking Googlebot.

Where do I start if I think I blocked Googlebot?

Check the CDN and WAF rule logs for Googlebot requests recorded as blocked or challenged, and turn on the verified bot allow setting. Then check in the Search Console Crawl Stats whether server errors are decreasing, and check changes in search visibility separately in the Search Console performance report.