"You can't edit robots.txt on Shopify" has been wrong since 2021. Create a template called robots.txt.liquid and you can edit it.

But that's not the interesting part. If you copy the code straight out of Shopify's own documentation to build that template, your store rolls back to an older ruleset than the one currently being served. No warning is shown.

To confirm this, we fetched /robots.txt directly from 6 live Shopify stores on September 1, 2026 and compared the output. Stores with no template and stores serving custom rules were clearly running on different base rulesets.

This article separates what Shopify actually lets you touch on robots.txt and sitemap.xml, what SEO apps genuinely handle on your behalf, and when the right move is to leave it alone — using primary sources and live measurements.

The short version

If you don't have a clear reason to change robots.txt, don't create robots.txt.liquid. The moment you do, you quietly regress to a ruleset older than the current default. That is the most practical finding of this research.

The sitemap.xml served by Shopify's Online Store cannot be customized — not with native features, not with apps. The generated file says so in a comment: "The sitemap can not be edited manually, but is kept up to date in real time." There are exactly two ways to keep a resource out of the sitemap: the seo.hidden metafield and the Unlisted product status.

Combining noindex with Disallow backfires. A page blocked in robots.txt is never crawled, so the noindex is never read. If anything links to it, it can stay in search results indefinitely.

Almost none of the major all-in-one SEO apps touch robots.txt. We checked every App Store feature tag: Yoast SEO, Smart SEO, Tiny SEO, SearchPie, and SEO Manager all lack the Robots.txt tag. Within the scope of this check, only two listings carried it — SEOWILL (formerly SEOAnt), an all-in-one app, and Robots.txt by DJP Digital, a single-purpose app.

And the Shopify Help Center section describing the default robots.txt does not match the real file as of September 1, 2026. More on that below.

How far can you actually go with robots.txt?

You have to build the theme template yourself

robots.txt.liquid is not included in default themes. You create it from Online Store > Themes > Edit code, then "Add a new template > robots". You cannot upload the file itself, and JSON templates are not supported.

What you can edit:

• Adding or removing Allow / Disallow rules for specific URLs
• Adding Crawl-delay
• Listing additional sitemap URLs
• Blocking or allowing specific crawlers (including AI crawlers)
• Branching rules by host (market) using request.host

Only six Liquid objects are available: robots, group, rule, user_agent, sitemap, and request. Ordinary global objects like shop are not accessible here.

Shopify explicitly calls this unsupported

The Help Center states it plainly: this is an unsupported customization, misusing it can cost you all of your traffic, and Shopify Support cannot help you with it.

There are two more operational traps:

1. robots.txt.liquid lives inside a theme. It has to be saved to your published theme, and switching themes switches your robots.txt with it
2. Uploading a theme through the admin's Themes page does not import robots.txt.liquid. It is preserved if you go through ThemeKit or the command line

In other words, a theme swap or a site redesign can silently wipe your robots.txt configuration.

The old-vs-new default gap, measured across 6 live stores

Here is the core of it. Shopify's official documentation offers this pattern for keeping the built-in rules while appending your own:

{% for group in robots.default_groups %}
  ...
{% endfor %}

The problem is that what robots.default_groups returns no longer matches the default robots.txt Shopify currently serves. Here is what we found by fetching /robots.txt directly from live stores on September 1, 2026.

Item	New generation (no template)	Old generation (template present)
Stores tested	deathwishcoffee.com / shop.polaroid.com / mini-mal.tokyo	allbirds.com / gymshark.com / hiutdenim.co.uk
Header comments	References UCP/MCP, agents.md, shop.app/SKILL.md	None from the default
Explicit Allow: lines	Yes (many)	None from the default
Disallow: /search	Absent (crawlable)	Present in all 3
Disallow: /policies/	Absent (crawlable)	Present in all 3
Disallow: /services, /sf_*	Present	Absent
Disallow: /cart.js	Present	Absent in all 3
Disallow: /recommendations/products	Present	Only on allbirds
Nutch / AhrefsBot / AhrefsSiteAudit / MJ12bot / Pinterest groups	Absent	Present in all 3 (with Crawl-delay)

All three stores emitting the old generation also carry custom rules that don't exist in any default (allbirds' /a/click*, gymshark's es-US entries, hiutdenim's Allow: /search/). Of these, allbirds and hiutdenim return powered-by: Shopify on the /robots.txt response along with a header indicating the Online Store robots template, confirming the file is being served from a theme template. gymshark is served through CloudFront, so we could not determine from the outside whether the output originates from a theme template.

Note that "None from the default" in the table means the header comments and Allow: lines present in Shopify's current new-generation default have been lost. Lines each store added on its own (gymshark's one-line comment, hiutdenim's Allow: /search/, and so on) still exist separately.

So: if you build the template "just to add one rule" the way the docs show you, you newly block /search and /policies/, and you lose the AI-agent header comments along with the entire set of Allow rules.

This mismatch was also reported to the Shopify Community on June 9, 2026 (Technical Q&A: "Robots.default_groups returns outdated rules, mismatch with the default robots.txt"). The differences the poster lists line up almost exactly with what we measured. The thread has two replies from ordinary users, but as of September 1, 2026 we could not confirm any response from official Shopify staff. The poster also writes that they contacted Shopify Support first and were directed to discuss it on the forum. Our testing backs up that report with live files.

The Help Center text doesn't match the real file either

The Shopify Help Center section on Shopify's default robots.txt lists Disallow: /search and Disallow: /policies/. Neither appears in the actual current default file.

shopify.dev has the same problem. The Liquid rule object reference page uses Disallow: /policies/ as its example rule, and the "Customize robots.txt" tutorial demonstrates "removing a default rule" by stripping out the /policies/ Disallow. Both still assume /policies/ is blocked by default.

Trust your own store's /robots.txt over the documentation. Open https://your-domain/robots.txt in a browser and you'll see exactly what is being served right now.

How far can you go with sitemap.xml?

Short answer: the Online Store sitemap cannot be customized

Shopify's sitemap.xml is generated and updated automatically. You cannot hand-edit it and you cannot swap in your own file. The comment at the top of the real file says as much: it can't be edited manually, and it's kept current in real time.

The structure is a parent sitemap referencing child sitemaps. As of September 1, 2026, these were the child files we observed (each is only emitted when the corresponding resource exists):

• sitemap_agentic_discovery.xml
• sitemap_products_1.xml
• sitemap_pages_1.xml
• sitemap_collections_1.xml
• sitemap_blogs_1.xml
• sitemap_metaobject_pages_1.xml (when metaobject pages exist)

That first one, sitemap_agentic_discovery.xml, is a child sitemap added for AI agents. Of the 6 stores we tested, it was present on all 4 where the Shopify Online Store serves the sitemap (deathwishcoffee / shop.polaroid / mini-mal.tokyo / allbirds). Notably, allbirds has sitemap_agentic_discovery.xml even though it emits an old-generation robots.txt, and it also serves /agents.md and /llms.txt. Sitemaps and AI-facing files are generated on a different layer from the theme's robots.txt.

gymshark and hiutdenim, by contrast, returned 404 for sitemap_agentic_discovery.xml, /agents.md, and /llms.txt. Both serve their sitemaps through something other than the Online Store — a CDN, or Hydrogen/Oxygen (hiutdenim uses a /sitemap/products/1.xml format; gymshark has a custom structure including hreflang_sitemap.xml). These are not Shopify's standard sitemaps. The "you can't customize it" constraint applies specifically to the sitemap.xml served by Shopify's Online Store. With a headless setup or a CDN in front, serving your own sitemap is possible (though Shopify does not recommend putting a proxy in front of a store and cannot support that configuration).

There are exactly two correct ways to exclude something

What you want	How	Scope of the effect
Make a product purchasable only via direct link	Set the product status to Unlisted	Excluded from sitemap, search engines, storefront search, Shopify Catalog, and Shop — plus collections, recommendations, and third-party sales channels
Remove from sitemap and search engines but keep it in collections and recommendations	Set the seo.hidden metafield (integer, value = 1)	Excluded from sitemap, search engines, storefront search, Shopify Catalog, and Shop. Stays in collections, recommendations, and third-party sales channels
Remove only from search results (staying in the sitemap is fine)	Conditionally output <meta name="robots" content="noindex"> in the <head> of theme.liquid	Stops indexing only. Does not remove it from the sitemap

If both are set, Unlisted wins.

Also note that seo.hidden can be applied to products, pages, and blog posts. Collections are not supported.

And there's one important fact here. seo.hidden is a Shopify system metafield, and the Help Center notes that third-party apps may use it to store data when hiding a resource from search engines and the sitemap.

The Help Center goes further: if you see the message that the namespace and key seo.hidden is already in use, that means a third-party app is using seo.hidden and you cannot edit it from the admin. In other words, some SEO apps' "noindex feature" and "sitemap exclusion feature" are, in practice, writes to this metafield. Knowing that, you can make the same change from the admin without installing anything. It also explains two risks: installing two apps of the same type causes collisions, and switching apps can leave the previous app's seo.hidden values behind.

Never combine noindex with Disallow

"I really want it gone, so I'll do both" produces the opposite of what you intend. A page blocked in robots.txt is never crawled, so the noindex is never read. If another site links to it, it can persist in search results. Google does not support noindex directives inside robots.txt.

Also, the "Indexed, though blocked by robots.txt" message in Search Console is normal behavior on Shopify. Shopify's own help documentation says that in most cases no action is needed.

What are SEO apps actually doing for you?

People tend to assume SEO apps on the Shopify App Store will manage robots.txt too. We checked every App Store feature tag to see whether that holds up.

App	robots.txt editing	Sitemap features	noindex	Starting price	Free plan	Japanese UI	BFS	Rating/reviews
Yoast SEO ‑ Store optimization	✗	✗	Listed	$19/mo	✗ (14-day free trial)	✗	✗	4.5 / 157
Smart SEO AI & Image Optimizer	✗	✓	✓ (including free plan)	Free / $14.99	✓	✓	✓	4.9 / 852
Tiny SEO Speed Image Optimizer	✗	✓	✓	Free to install / $14	✓ (usage-based)	✓	✓	5.0 / 2,284
SearchPie: SEO, Speed & Schema	✗	✓	✓	Free / $39	✓	✓	✓	4.9 / 2,355
SEO Manager | venntov	✗	✓	✓	$9/mo	✗ (7-day free trial)	✗	✗	4.7 / 389
SEOWILL: AI SEO & AI Blog Post (formerly SEOAnt)	✓	✓	Not listed	Free / $29.99	✓	✓	✓	4.8 / 1,741
Robots.txt (DJP Digital)	✓ (dedicated)	✗	✗	$29 one-time	✗	✗	✗	5.0 / 3
LLMs, Robots and Agents Editor	No feature tag (support stated in the description)	✗	✗	Free	✓	✗	✗	5.0 / 2

All data as of September 1, 2026; prices in USD. The "robots.txt editing", "Sitemap features", and "noindex" columns are judged by the presence of the App Store listing's feature tags (Robots.txt / Sitemaps / Page indexing). Tiny SEO's entry-level plan is "Free to install": up to 50 image optimizations per month are free, with usage-based charges beyond that.

Three things stand out.

First, the all-in-one SEO apps with the most reviews don't handle robots.txt at all. Five out of five lack the Robots.txt feature tag. What they do provide is meta tags, structured data, image optimization, 404 redirects, and noindex/sitemap exclusion that most likely runs on seo.hidden. Editing robots.txt is simply outside their scope.

Second, the apps that explicitly do handle robots.txt have almost no track record. The dedicated app Robots.txt (DJP Digital) has 3 reviews; LLMs, Robots and Agents Editor has 2. That's too few to read any pattern from. The two reviews on LLMs, Robots and Agents Editor are one 5-star and one 1-star, the 1-star describing an Application Error where the app failed to launch immediately after install (the developer replied that it stemmed from a dependency update and has been fixed). Note that despite that split, the listing still displays a 5.0 rating, which suggests the aggregate hasn't caught up.

Third, that same LLMs, Robots and Agents Editor addresses the exact problem in this article head-on in its product description. The listing explains that Shopify auto-generates robots.txt, llms.txt, and agents.md, that customizing them requires editing a Liquid theme template, and that doing so risks breaking rules Shopify updates automatically. It then claims the app edits while preserving dynamic rules, provides diffs, validation, and restorable backups for every change, and alerts you when the theme changes or when Shopify changes its rules (Radar).

That said, while the app is free, it supports only Spanish and English, and the listing states plainly that the developer does not offer direct support in English. The developer is Pholidota Studio SLU, based in Madrid, Spain. There is no Japanese UI.

The app is also listed under the App Store category "Site optimization - Other" and carries no SEO-category feature tags. That means it has no Robots.txt tag at all — its robots.txt support is something we confirmed from the listing copy, not the tags. Searching by feature tag will not surface it.

As for SEOWILL's Robots.txt feature, we only confirmed that the App Store feature tag exists. We could not confirm from official sources how it is actually implemented — whether it generates a template, or whether it addresses the robots.default_groups regression. You'd need to ask the developer before installing.

Decisions by scenario

You have no clear reason to change robots.txt

Do nothing. That's the safest position. Shopify updates the default content on an ongoing basis, and creating a template cuts you off from those updates.

You want to block specific AI crawlers or suppress crawling of custom parameters

If you have Liquid expertise in-house or a Shopify Partner you can hand this to, build robots.txt.liquid yourself. But follow these steps without exception:

1. Before creating anything, copy and save the full current /robots.txt
2. After creating the template, diff the old and new output
3. Manually restore the differences (/search, /policies/, the Allow: block, the header comments)
4. Save it to the published theme, and re-check whenever you switch themes

Also note that blocking AI crawlers in robots.txt does not stop your products from being distributed to agentic storefronts via Shopify Catalog. That's a separate control layer.

You want a one-off change without adding a monthly fee

Robots.txt (DJP Digital) is a $29 one-time purchase. You do have to weigh the thin track record of 3 reviews. The listing also states that no theme changes are required, but we could not confirm from public information how that reconciles with Shopify's own rule that robots.txt can only be customized through a theme template.

You want to keep specific pages out of the sitemap

You don't need an app. The Unlisted status or the seo.hidden metafield covers it. If you already run an SEO app, its noindex feature may be using the same metafield, so be careful not to set it twice.

You run multiple languages or countries with Shopify Markets

hreflang, canonical tags, multilingual sitemaps, and crawler exemption from automatic redirects are all handled automatically. Leave them alone. Automatic hreflang is enabled by default on every store, and automatic redirection applies only to customers — known search engine crawlers are excluded. Only if your theme or an app is emitting its own hreflang should you turn off automatic hreflang under Online Store > Preferences ("Social sharing image and SEO") to avoid duplication.

Use request.host in robots.txt.liquid only when you genuinely need per-domain crawl control.

Pre-flight checklist

1. Have you opened your own store's /robots.txt in a browser and looked at the real thing?
2. Is robots.txt.liquid already installed? (If so, you may be on the old generation)
3. Is robots.txt genuinely the only way to achieve the change you want?
4. Have you saved the full /robots.txt before making changes?
5. Have you diffed after the change and confirmed there are no unintended differences?
6. Are you planning a theme switch? (Can you build re-checking robots.txt into that process?)
7. For sitemap exclusion, are you using seo.hidden or Unlisted rather than noindex?
8. Are you combining noindex and Disallow anywhere?
9. Is an installed SEO app already using seo.hidden? (Check for collisions and leftovers)
10. If you use Markets, are you adding hreflang yourself?

Summary

Work through it in this order.

1. Don't touch robots.txt unless you have a clear reason to. Build the template the way the docs show you and you regress to rules older than the current default
2. The sitemap.xml served by the Online Store cannot be changed — not natively, not with apps. Exclusion is a two-way choice: seo.hidden or Unlisted
3. Never combine noindex with Disallow. The noindex on a blocked page is never read
4. Don't expect all-in-one SEO apps to manage robots.txt. None of the five major apps do. If you need it, the choice is hiring a partner or accepting the risk of a dedicated app with a very thin track record

One thing worth stating clearly: where a store is based makes no difference here. Across our tests, the default robots.txt on mini-mal.tokyo, deathwishcoffee.com, and shop.polaroid.com was identical apart from a single line containing a store-specific ID. The sitemap.xml structure was identical too, differing only in whether the relevant resources (products, metaobjects, and so on) existed. The variables that actually change your output are on your side of the store: Markets, theme, and apps.

The default content went through a significant generational change during 2026 and may change again. Everything here reflects what we verified in live files. Researched September 1, 2026. Before you make any change, check your own store's /robots.txt directly.