faceted navigation

On a store with categories times colors times sizes times price bands, the math runs away fast. A few filters multiplied across a catalog can balloon into millions of URL combinations, and Google notes that crawlers “can’t determine whether the URLs are going to be useful without crawling first.” Every request spent on ?color=blue&sort=price is a request not spent finding your new products.

Deciding what to index

Not every filtered page is waste. A combination with real search demand, say “red running shoes”, can earn its own indexable landing page. Presentation-only variants (sort order, session IDs, low-stock permutations) should not.

Google’s documented options for controlling this:

• robots.txt disallow for filter URLs you never want crawled (sort, session, price sliders).
• rel=canonical pointing filtered pages back to the clean category, which over time lowers crawl volume on the non-canonical versions.
• rel=nofollow on links to filtered pages, so crawlers are less likely to discover them.
• HTTP 404 when a filter combination returns no products.
• The & separator for parameters, since commas, semicolons, and brackets are hard for crawlers to detect as separators.

A noindex, follow on thin combinations you still want crawled for link flow is a common practitioner approach too, though Google’s own faceted-navigation guide does not list it among its methods.

Get the mix wrong and you either bloat the index with duplicates or block pages that could have ranked. AI answer engines add another reason to keep this tidy: they rely on crawling too, so pages buried under filter noise are less likely to be fetched and cited.