# Magento faceted navigation and crawl budget

Stop Magento filter URLs wasting crawl budget. Canonicals, robots, AJAX filters and indexable facets.

- Date: 2026-08-10
- Category: Magento

Magento faceted navigation is the usual reason a healthy catalogue still wastes crawl budget. Filters help shoppers. They also mint URL combinations Google will fetch forever if you let Magento publish every colour, size, price and stock stack as an indexable page. This note is for Magento 2 and Adobe Commerce stores with layered navigation on, a large attribute set, and a Search Console crawl graph that looks like a Christmas tree.

I am Alan Vo. I have spent eighteen years on storefronts, including Magento work where category filters were the merchandising system, not a widget. Adobe documents the admin switches in [layered navigation](https://experienceleague.adobe.com/en/docs/commerce-admin/catalog/navigation/layered-navigation). The docs do not tell you which facets should ever rank. That is the production job.

## How Magento faceted navigation burns crawl budget

Magento faceted navigation burns crawl budget when every filter combination becomes a crawlable URL, often with a self-canonical, a sitemap entry, or both. Googlebot then spends its time on `/curtains.html?color=blue&lining=thermal` instead of the category and products you actually wanted indexed.

Magento’s layered navigation is attribute-driven. You mark an attribute as filterable, assign it to a category, and the layered block renders links. Older themes output full page loads with query parameters. Some modules rewrite those into path URLs that look like new categories. AJAX filters can hide the URL from the address bar and still leak a parameter when someone shares a link. All three patterns exist in the wild. All three can explode.

Crawl budget is not a myth on a 20,000 SKU Magento store. Google will crawl. It will not crawl everything equally. If half the requests in Search Console crawl stats are parameter URLs, your product pages wait. That shows up as “Discovered - currently not indexed” on SKUs that merchandisers swear are live.

Duplicate content is the twin problem. The same 24 products appear on the category, on colour, on colour plus size, and on a sort of price. Titles and H1s stay identical or get a clumsy suffix. Magento’s canonical setting may point filter URLs at the clean category, which is often correct, or at themselves, which tells Google each combo is a unique document. Self-canonicals on thin filter pages are how you get an index full of junk.

I also see `Disallow: /*?` used as a blunt instrument. It stops Googlebot fetching query strings, which can be right, and it can hide pagination, sorting, and campaign URLs you still needed crawled. Robots.txt is not `noindex`. A disallowed URL can still appear in results without a snippet if it is linked. Facets need a policy, not a single disallow line copied from a 2016 blog.

![Magento faceted navigation catalogue work on a large monitor](../../images/blog/magento-faceted-navigation-1.jpg)

## Why Magento filter URLs still matter in 2026

They still matter because shoppers search the way they filter, and AI crawlers now hit the same HTML, while Magento stores keep adding attributes. “Blue made-to-measure curtains” is a real query. A dedicated, indexable facet can earn that. A thousand accidental combinations cannot.

What changed is the cost of getting it wrong. Crawl waste used to annoy SEO reports. Now it also starves product schema and category copy that AI answers might cite. If Googlebot never reaches the PDP, neither does a retrieval bot that follows the same links.

Internal linking is part of the same system. Facet links in the sidebar are internal links. If they all pass equity into parameter URLs that you then `noindex`, you have built a maze. The [internal linking notes for ecommerce catalogues](https://alanvo.com/blog/ecommerce-internal-linking/) cover hub pages and breadcrumbs. Facets are the Magento-specific version of that maze.

Custom product stores make this sharper. Made-to-measure and configurators already have option URLs. Layered navigation on top of that is how you create a crawl graph nobody can draw. I want filters for UX. I want a short list of indexable landings for SEO. Those are different lists.

## How Magento faceted navigation should work in production

Magento faceted navigation should work as shopper UI first, with a small set of crawlable facet landings, and everything else `noindex` or not a URL. The admin “filterable” flag is not an SEO flag.

I start by listing attributes that are filterable in layered navigation. For each one I ask whether it has standalone search demand and enough products to make a page. Brand, a product type, a material, a size family: maybe. Price slider, “new”, “sale”, arbitrary boolean flags, and stacked combinations: almost never.

Then I look at how URLs are built.

Query parameters: `?color=red`. Easy to `noindex` with a robots meta or to canonicalize to the category. Easy to disallow. Bad as a ranking URL unless you rewrite them.

Rewritten paths: `/red/` under a category. These look like real pages. They need unique titles, copy, and a canonical strategy. If the module creates `/red/large/thermal/` automatically, you are back to combinatorics. Limit depth to one facet, two at most, and only for pairs you chose.

![Warehouse shelves standing in for a Magento inventory grid](../../images/blog/magento-faceted-navigation-2.jpg)

AJAX with history API: the shopper sees a parameter, Google may or may not. If the filter state is only in JS, Googlebot may see the unfiltered category, which is often what you want. If the history URL is crawlable and self-canonical, you accidentally published thousands of pages again.

Canonicals must be consistent with sitemaps. If the sitemap lists only clean categories and products, filter URLs should canonical to those, or be `noindex`. If you want a facet to rank, put it in the sitemap, give it copy, and do not canonical it away.

Pagination and sort must be excluded from the facet discussion. `?p=2` and `?product_list_order=price` are not landings. Canonical them to the category or to the same page without sort. Magento’s toolbar is a frequent source of duplicate titles.

I use `noindex,follow` on parameter URLs I still need humans to open, and robots.txt disallow only for patterns that should not be fetched at all, such as session IDs and `___store=`. I do not hide CSS and JS Magento needs to render the category, or Googlebot will see an empty layout and you will debug “why is my PLP not indexed” for a month.

The [Magento SEO audit checklist for 2026](https://alanvo.com/blog/magento-seo-audit-2026/) is the parent process. Facets are the layer that usually fails first.

## Production checklist for Magento facets

1. Export every attribute with “Use in Layered Navigation” enabled. Mark each as indexable landing, `noindex` filter, or remove from layered nav.

2. Crawl a single large category with filters followed. Count unique URLs. If the number is combinatorial, you do not have a content strategy. You have a generator.

3. Inspect canonicals on: clean category, one single-facet URL, one two-facet URL, one sort URL, one pagination URL. They should not all canonical to themselves.

4. Compare XML sitemaps to the crawl. Sitemaps should not include junk facet URLs. Magento’s sitemap generator plus a layered-nav module often disagree.

5. Set meta robots for non-indexable facet URLs in the module or via a plugin. Do not rely on merchants remembering to tick a box per category.

6. Write unique titles and an H1 plus a short intro only for the facet URLs you chose to index. Thin auto-generated pages are why Google leaves them out.

7. Keep AJAX filters from creating crawlable history URLs unless those URLs are in the indexable set.

![Boxed products on a packing bench](../../images/blog/magento-faceted-navigation-3.jpg)

8. Block session IDs, `SID=`, `___store=`, and internal Magento routes in robots.txt. Do not use a global `Disallow: /*?` until you have listed exceptions for anything you still need crawled.

9. Make sure category internal links in menus and body copy point at the clean URL, not at last week’s filtered state copied from a browser bar.

10. Test store views. A filter URL that works in one view and 404s in another still gets crawled.

11. After deploy, purge full-page cache on a sample of facet URLs. Magento will serve a stale canonical longer than anyone expects.

12. Watch Search Console crawl stats weekly for a month. The share of parameter URLs should fall. Indexed product count should not fall with it. If products drop, you over-blocked.

## What breaks when you lock down Magento faceted navigation

Merchandising breaks first if you remove filters shoppers use. SEO lock-down is not “turn off layered navigation”. It is “layered navigation may not create documents”. Keep the UX. Change the index.

Paid search landing pages break if they used a query string Magento now disallows. Coordinate with ads. Give campaigns clean category or CMS URLs.

Facet landings you wanted can break if a developer canonicals every parameter to the parent. You cannot rank “linen curtains” on a URL that tells Google it is really just “curtains”. Pick the URLs that deserve to be documents.

AJAX-only filters can break crawlable internal linking. If Google only sees the unfiltered category, that is fine for budget, and it means your only path to a product is category, search, or sitemap. Make those strong.

Multilingual stores break when a facet rewrite is not translated and hreflang points at a 404. Facet modules are often English-first.

Performance can break in the other direction. Some “SEO filter” modules render huge link lists of every combination in the footer. That is not SEO. That is a 2MB HTML gift to nobody. INP and LCP suffer. I strip those blocks.

## How to measure Magento faceted navigation

You measure Magento faceted navigation by crawl waste, index composition, and whether the chosen facet landings actually rank, not by how many filter links exist in the sidebar. Search Console crawl stats and a crawler export are the instruments.

Look at: requests to URLs with `?` versus clean paths; number of indexed pages that are facet URLs; impressions on those URLs; and product coverage. A successful lock-down reduces facet URLs in the index and does not reduce product URLs.

![Watch product still used as a Magento PDP example](../../images/blog/magento-faceted-navigation-4.jpg)

For the few indexable facets, track queries, not vanity index counts. If “blue curtains” still lands on the parent category after you built `/curtains/blue/`, the title, H1 and internal links are wrong.

Compare before and after with the same crawl settings. Magento staging is useless here. Parameter behaviour on production cache is the truth.

I do not invent a conversion rate for turning off junk URLs. The win is crawl going to SKUs. On Magento contract work with deep option trees, that is also a usability win: shoppers can filter without Google indexing the unfinished combinations.

## Related Magento work on this site

Faceted navigation is where made-to-measure and configurator catalogues live or die. I shipped Magento storefronts as contract development for [Direct Fabrics](https://alanvo.com/work/direct-fabrics/), a made-to-measure curtains and blinds shop where spec options have to stay simple, and for [schrankwerk](https://alanvo.com/work/schrankwerk/), where a cabinet configurator is the catalogue. Those sites need filters and options. They do not need Google to crawl every unfinished combination. That is the same rule I use on any Magento layered nav audit.

## Magento faceted navigation FAQ

### Should Magento faceted navigation URLs be indexed?

Magento faceted navigation URLs should be indexed only when a single facet (or a rare, chosen pair) has search demand and unique page content. Stacked filters, price sliders and sort states should not be documents. Index the category and the products first.

### Does Magento faceted navigation require AJAX filters for SEO?

Magento faceted navigation does not require AJAX for SEO. AJAX can hide junk URLs if you do not push crawlable history states. Query parameters with a parent canonical can also be fine. The failure is crawlable, self-canonical thin combos.

### Can robots.txt fix Magento faceted navigation on its own?

Robots.txt cannot fully fix Magento faceted navigation. It can stop fetching patterns. It cannot add uniqueness, and a disallowed URL can still appear in results if it is linked. Use canonicals, `noindex`, sitemaps and a short indexable facet list together.

### How do I keep Magento faceted navigation usable after SEO changes?

Keep Magento faceted navigation usable by leaving the filter UI in place and changing only whether those states are URLs in the index. Shoppers can still narrow colour and size. Google should not receive a new document for every click.


![Retail aisle used while thinking through Magento filters](../../images/blog/magento-faceted-navigation-5.jpg)


HTML version: https://alanvo.com/blog/magento-faceted-navigation/