What Is Crawl Budget?

Two Forces Set It.

Crawl capacity is what your server can serve without degrading: a slow or error-prone site gets crawled less to avoid harming it. Crawl demand is how much the engine wants your pages, based on how often they change and how valuable they appear.

Technical search specialists analysing crawl budget metrics and search crawler behaviour — Optimizing Crawl Budget ensures complete search engine discoverability and sustainable organic rankings.

For most sites it is not a real constraint.: A few hundred pages that respond quickly will be crawled thoroughly. It becomes a genuine issue at scale, or where a site generates large numbers of near-identical URLs: faceted filters, session parameters, endless pagination, calendar pages that continue forever.
When it does bite, the symptom is quiet: new pages take a long time to appear
updates take a long time to register
Crawl statistics showing effort spent on parameter URLs rather than real content are the usual confirmation.
The fix is almost always removing waste rather than requesting more.: Stop generating low-value URLs, resolve redirect chains, return correct status codes, and keep the pages you care about reachable in few clicks from a page that is crawled often.

Covered in Technical SEO

Frequently Asked Questions

Does Our Small Site Need To Worry About Crawl Budget?

Almost certainly not. A site of a few hundred healthy pages is crawled comfortably. Crawl budget becomes a real constraint at large scale, or where filters and parameters generate far more URLs than there is content.

Can We Increase Our Crawl Budget?

Not directly. You can make the site faster and more reliable, which raises capacity, and remove low-value URLs so the available crawling is spent on pages that matter.

What Does This Cover?

Crawl budget is the number of URLs a search engine is willing and able to fetch from a site in a given period. It is set by how much crawling the site can tolerate and how much the engine considers worthwhile.

Connected

Where this applies

Related Services

Technical SEO

All Resources

Crawl Budget

Two Forces Set It.

Frequently Asked Questions

Where this applies

Related Services

Read next

Book a consultation

Your Details

Consultation Preference

Crawl Budget

Two Forces Set It.

Frequently Asked Questions

Where this applies

Related Services

Read next