# Magento robots.txt for AI crawlers in 2026

Default Magento robots.txt often blocks AI search bots. Allow search crawlers, still hide checkout and filters.

- Date: 2026-07-27
- Category: Magento

Magento robots.txt in 2026 is still the file that decides whether Googlebot, Bingbot and AI search crawlers may fetch your catalogue. Magento 2 and Adobe Commerce generate that file from admin, often with a wildcard `User-agent: *` and a list of Disallows written for 2015. Those rules hide checkout, which is good, and they can hide the whole store from a bot that only matches the wildcard, which is not. This note is for anyone who has not opened `/robots.txt` on production since go-live.

I am Alan Vo. Eighteen years of storefronts, including Magento catalogues, taught me to read robots before I rewrite titles. Google’s own [introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) is the rulebook: robots.txt manages crawling. It is not a secrecy tool. A disallowed URL can still appear in results if something links to it. Use `noindex` or authentication when you truly need a page out of the index.

## Why default Magento robots.txt hurts AI search

Default Magento robots.txt hurts AI search when a wildcard Disallow, a staging leftover `Disallow: /`, or a CDN-cached old file is the only rule AI crawlers inherit. Magento’s admin default is meant to keep bots out of `/checkout/`, `/customer/`, `/catalogsearch/`, and a pile of internal routes. That intent is still correct. The damage is order, wildcards, and files nobody re-read after AI crawlers showed up with their own user-agent names.

Magento does not ship a static `robots.txt` in the web root the way a brochure site does. Content, Design, Configuration, Search Engine Robots holds the custom instructions. Cloud projects often generate the response from the database and may serve it through Fastly from `pub/media`. You can “save” in admin and still watch the edge cache serve last month’s Disallow list. I fetch the live URL with curl every time.

AI search crawlers in 2026 are not one bot. Training crawlers and retrieval crawlers are different user-agents. A store that wants to appear in ChatGPT or Perplexity answers needs the retrieval bots to be allowed to fetch product and category HTML. A store that only wants Google can still allow Googlebot and disallow training crawlers. That is a policy choice. What you cannot do is assume Magento’s `User-agent: *` block was written with those names in mind.

The other Magento-specific trap is `Disallow: /*?` or aggressive query-string blocks. They exist because faceted navigation explodes. They also block any bot, including AI bots, from fetching a URL with a parameter you actually needed. Pair robots with the facet policy in the [Magento SEO audit](https://alanvo.com/blog/magento-seo-audit-2026/), not as a one-line panacea.

![Magento robots.txt catalogue work on a large monitor](../../images/blog/magento-robots-txt-ai-1.jpg)

WAF rules sit beside robots. I have seen Magento Cloud and generic WAFs challenge unknown bots with a JS interstitial. Googlebot may be allowlisted. OAI-SearchBot then gets a challenge page. robots.txt says Allow. The WAF says no. The audit is incomplete if you only read the text file.

## Why Magento robots.txt is an SEO ticket in 2026

Magento robots.txt is an SEO ticket in 2026 because AI answers now fetch pages, and Magento’s generated file is still the first gate. Titles and schema do not matter to a bot that never received the HTML.

What changed is the bot list, not Magento’s need to hide cart and account. Checkout, customer, wishlist, catalog search dumps, API routes, and `___store=` parameters should stay disallowed. Product, category and CMS HTML should stay allowed for the crawlers you want.

llms.txt is a different file. It is a curated map for agents, not a crawl permission file. I cover that in [llms.txt and AI search visibility](https://alanvo.com/blog/llms-txt-ai-search/). You can publish a beautiful llms.txt and still block every AI crawler in robots.txt. They do different jobs. Do both or neither, but do not confuse them.

Google’s limitation still applies: robots.txt is not `noindex`. If you need a Magento CMS page out of search results, use meta robots or remove it. If you need it private, put it behind login. Disallowing `/customer/` is correct. Disallowing `/` on production because someone copied staging config is a Sev-1.

## How to write Magento robots.txt that still hides checkout

You write Magento robots.txt by allowing the crawlers you want on catalogue HTML, then listing Magento routes that must never be crawled, with AI user-agent blocks only where you intend them. Put named Allow groups where your policy needs them. Keep `User-agent: *` for the Magento path list. Do not put `Disallow: /` on production.

I start with policy in one sentence: retrieval bots may fetch category, product and content HTML. Training crawlers may or may not, depending on the merchant. Googlebot and Bingbot may. Checkout, cart, customer, search result dumps, compare, wishlist, REST, GraphQL, and admin-like paths may not.

Then I look at the live file. If I see `Disallow: /` under `User-agent: *`, I treat production as blocked until proven otherwise. Staging leftovers survive Magento content deploys more often than people expect.

![Warehouse shelves standing in for a Magento inventory grid](../../images/blog/magento-robots-txt-ai-2.jpg)

Magento path list I still want under the wildcard, adjusted per store:

- Checkout, cart, customer account, wishlist
- Catalog search and advanced search result URLs
- Product compare
- Send friend and review write routes if they are form endpoints, not the public review list you meant to index
- `SID=` and `___store=` parameter patterns
- `/lib/`, `/var/`, `/setup/` class paths if they are web-reachable (they should not be)
- REST and GraphQL if they are public on the same host

I do not Disallow CSS, JS, or `pub/static` that Magento needs to render. Google has been clear that blocking resources hurts rendering. Magento pages that look empty to Googlebot are often over-blocked assets, not a content problem.

I do not put the admin path in robots.txt. That advertises it. Restrict admin by URL, IP allowlist, and auth. Robots.txt is a public document.

For AI crawlers, I name the user-agents the merchant cares about and either Allow catalogue paths or Disallow `/` for training-only bots. I keep that list on a quarterly calendar because names change. I do not copy a 40-bot blocklist from a random GitHub dump onto a Magento shop without reading it. Over-blocking Bing’s crawler because a list aliased something badly is a real outage.

Sitemaps: Magento should advertise the sitemap index in robots.txt. `Sitemap: https://www.example.com/sitemap.xml` with the real host. Wrong protocol or store-code hosts in that line waste a year.

Cloud cache: after admin save, purge the edge cache for `/robots.txt` and `/robots`. Verify with curl against the production host, not an internal admin preview.

llms.txt does not replace these rules. If you add llms.txt, still allow the markdown and HTML those links point at.

## Production checklist for Magento robots.txt

1. Open `https://your-host/robots.txt` in a browser and with curl. Confirm it is not a 404, a homepage, or a login wall.

2. Search the file for `Disallow: /` on its own. If it exists under a wildcard on production, fix that before any other SEO work.

3. Confirm checkout, cart, customer and catalogsearch are disallowed. Confirm category and product paths are not.

4. Decide retrieval vs training crawlers in writing with the merchant. Implement named user-agent groups that match that policy.

5. Add the sitemap index line with the canonical host.

6. Remove any Disallow of static assets Magento needs to paint PLP and PDP.

7. Do not mention the admin front name.

![Boxed products on a packing bench](../../images/blog/magento-robots-txt-ai-3.jpg)

8. Align query-string Disallows with the facet strategy. A global `/*?` needs a review, not a reflex.

9. Purge CDN cache for robots.txt. Recrawl with Google Search Console robots tester.

10. Check WAF and bot-fight settings for the same user-agents. Robots Allow plus a JS challenge is still a block.

11. Keep staging on `Disallow: /` or `noindex`. Keep production off that setting. Magento config scopes make this a website-level landmine.

12. Re-read the file after every Magento upgrade and every SEO module install. Modules love to append rules.

## What breaks when you edit Magento robots.txt

Faceted navigation sales do not break. Crawl of junk filter URLs might stop, which is the point. If you over-Disallow, Google stops fetching paginated categories you still needed. Watch the Pages report.

AI visibility breaks if you Disallow retrieval bots by accident while cleaning training bots. Name them. Do not rely on “the wildcard will be fine”.

Locales break when robots.txt is website-scoped but you have multiple domains. Magento can serve one robots file for several stores if config is wrong. Each host should speak for itself.

Adobe Commerce Cloud breaks when Fastly keeps the old file. The admin looks right. Google’s tester looks wrong. Purge.

App crawlers and uptime monitors break if you Disallow `/` for unknown agents too aggressively. That is rare. More common is blocking Googlebot by putting it in a Disallow group you copied.

Security theatre: people add `Disallow: /admin`. Attackers read robots.txt. Change the admin path and lock it down instead.

## How to measure Magento robots.txt changes

You measure Magento robots.txt changes with Search Console robots tester, crawl stats, a sample fetch as Googlebot, and whether AI retrieval bots receive 200 HTML on a PDP, not with a homepage Lighthouse run.

After a change, request indexing on a product URL and watch crawl stats for a drop in parameter URLs if that was the goal. Indexed product count should hold.

For AI, fetch a PDP with the user-agent string of the retrieval bot you allowed. If you get 403, 429, or a challenge HTML, robots.txt is not your only problem.

I do not promise traffic from “opening ChatGPT’s crawler”. I promise the file will stop lying. On large Magento catalogues, that is already the difference between Google spending budget on checkout echoes and spending it on SKUs.

## Related Magento work on this site

Robots.txt is crawl policy for real Magento catalogues, including the [retail conversion Magento stores](https://alanvo.com/work/retail-conversion/) where I directed ecommerce and SEO, and storefronts like [Egaming](https://alanvo.com/work/egaming/), a bespoke Magento theme for a digital games catalogue with filters at scale. Games shops and appliance shops both mint URLs. Both need checkout hidden and products fetchable. That is the Magento robots job.

![Watch product still used as a Magento PDP example](../../images/blog/magento-robots-txt-ai-4.jpg)

## Magento robots.txt FAQ

### Does Magento robots.txt block AI crawlers by default?

Magento robots.txt can block AI crawlers by default when they inherit a restrictive `User-agent: *` group, a leftover `Disallow: /`, or a query-string ban that hides the URLs they would fetch. It is not that Magento names GPTBot in core. It is that the wildcard was written for Google-era Magento paths and never revisited.

### Should Magento robots.txt allow every AI bot?

Magento robots.txt should not allow every AI bot by reflex. Allow retrieval crawlers if you want AI search citations. Disallow training crawlers if that is the merchant’s policy. Keep Googlebot able to fetch catalogue HTML either way.

### Can Magento robots.txt hide checkout from Google?

Magento robots.txt can stop Google fetching checkout, which is what you want for crawl budget. It will not reliably hide a URL that is linked from elsewhere. Checkout should also be `noindex` and not in sitemaps. Robots is the crawl valve, not a lock.

### Where do I edit Magento robots.txt?

You edit Magento robots.txt in admin under Content, Design, Configuration, Search Engine Robots for the correct website scope, then you verify the public URL and purge CDN cache. On Adobe Commerce Cloud, confirm the generated file is the one Fastly serves. Do not only edit a file on disk that Magento is not using.


![Retail aisle used while thinking through Magento filters](../../images/blog/magento-robots-txt-ai-5.jpg)


HTML version: https://alanvo.com/blog/magento-robots-txt-ai/