Where they started
A marketplace with a healthy 8,000-product catalog and a link profile most competitors would envy, yet new category pages took three to four weeks to appear in Google. The cause was in the log files: faceted navigation generated a fresh URL for every filter combination, and the site was serving ~412,000 crawlable URLs off those 8,000 products. The worst single filter path stacked eleven parameters before it dead-ended (color, then size, then price band, then material, then five more), each a URL Googlebot dutifully crawled. Crawl budget was being burned on filter permutations no shopper would ever land on, while genuinely new pages sat in the queue behind them. This was not a content problem or a links problem. The crawler simply never got there in time.
What we did
- Wrote a facet indexation policy, not a blanket block. Filters with real search demand (a handful of color and category combinations) became indexable landing pages; everything else got canonicalized to its parent and closed off from crawling. Cutting noise without killing the pages that earn traffic was the whole balance.
- Enforced the policy where crawlers actually decide. Canonicals for near-duplicates, robots directives for the combinatorial tail, and parameter handling so the 400k+ junk URLs stopped competing for budget. Crawlable URLs fell to ~19k.
- Verified against server logs, not guesses. We tracked Googlebot’s real hit distribution before and after. Watching budget shift off filter paths and onto categories and products confirmed the fix landed, weeks before rankings moved.
- Built links, but modestly. 28 referring domains over nine months. On this site they were a supporting move; the constraint was crawl reach, and links do not fix crawl reach.
The curve stayed flat for two months while the crawler re-mapped the site, then climbed steadily once budget reached the pages that deserved it. The honest caveat: a pure-technical win this clean only happens when crawl is genuinely the bottleneck. On most sites it is not, and no amount of parameter-tidying substitutes for content and authority. Here it was, so we fixed it.