Search engines do not fetch every page of every site continuously. They allocate a rate of requests per site, and on a large site that allocation becomes a real constraint on what gets indexed.
The allowance is set by capacity and by interest
Two separate factors determine how much crawling a site receives. The first is how much load the server appears able to absorb without degrading, which the crawler probes and adjusts against.
The second is how much the search engine expects to gain from fetching more, which is inferred from how often content changes and how much of it has proved worth showing to users.
A slow server reduces the first factor directly, and a site full of pages nobody selects in results reduces the second. Both effects compound on sites with very large page counts.
Faceted navigation multiplies pages without adding content
Filter and sort parameters on a catalogue can generate an enormous number of distinct addresses, each returning a slightly different arrangement of the same items.
Crawlers follow those links and spend their allowance discovering combinations that no one searches for and that duplicate each other almost entirely.
The pages that matter commercially then wait longer to be recrawled, which delays how quickly price changes, stock status and new products appear in results.
Redirect chains and error pages consume the same allowance
Every fetch counts, including the ones that end in a redirect or a failure. A chain of three redirects consumes several requests to deliver one page.
Sites that have been through multiple migrations often carry thousands of such chains, quietly absorbing crawl capacity that could be spent on live content.
Soft errors are worse, because a page returning a success code with no useful content is fetched repeatedly and never recognised as something to stop requesting.
Signals guide crawlers rather than commanding them
Directives that block crawling prevent the fetch but also prevent the engine from seeing anything on the page, including instructions about which version to prefer.
Canonical tags work differently, allowing the fetch while consolidating the ranking signals onto one preferred address. They are treated as strong hints rather than orders.
Using the wrong instrument is common. Blocking a page that carries a canonical tag hides the tag, leaving the engine to guess at exactly the point the site was trying to be clear.
The problem is concentrated at the top of the size range
Small sites rarely encounter this at all, because their entire page count fits comfortably inside any reasonable allowance.
The constraint bites on catalogues, classifieds, marketplaces and publishers with deep archives, where the number of addresses runs well beyond what commercial value justifies.
The remedy on those sites is reduction rather than optimisation. Fewer addresses, resolved cleanly, produce faster indexing than any attempt to increase the allowance itself.