Crawl Budget, Explained Like I'm Five
The technical SEO concept that finally clicked once I stopped reading definitions and just built a crawler.
1 min readSearch & AI
Imagine Google is a librarian. Every day, this librarian visits your library (your website) to check for new books (pages). But the librarian is busy — she can only visit so many libraries a day, so she allocates a "budget" of time to each one.
That's crawl budget. It's the number of pages Googlebot will crawl on your site in a given timeframe. If your library has 10,000 books but the librarian only checks 100 per day, new books might take months to be discovered.
The fix isn't just "make your site faster" — though that helps. It's about making sure Google spends its limited time on the pages that actually matter. Prune low-value URLs. Fix redirect chains. Make sure your sitemap is clean. Use noindex on pages you don't need in the index.
Building my own crawler made this click. Watching it waste time on duplicate pages and broken links made me viscerally understand why Google cares about this stuff. It's not about being punitive — it's about efficiency at a scale most developers never think about.