Search & AI
The Search & AI panel controls how search engines and AI crawlers see your site. From one screen you decide whether the site is indexed, whether AI bots may crawl it, and you maintain an llms.txt — a curated overview for AI assistants.
Where the module lives
Open Search & AI from the admin sidebar (/admin/seo). The panel has three controls — a website-indexing dropdown, an AI-crawler toggle, and an llms.txt editor. Click Save to apply all three at once.
Website indexing
The Website indexing dropdown decides whether search engines and AI may index the whole site:
| Option | Effect |
|---|---|
| Indexed (search engines + AI allowed) | Normal operation — the site is crawlable and indexable. This is the default. |
| Not indexed (staging / hidden) | The site is hidden site-wide: robots.txt returns Disallow: / and every page sends <meta name="robots" content="noindex, nofollow">. |
"Not indexed" is for staging, not live sites
Use Not indexed for a pre-launch or staging site you don't want in search results. On a live site it removes you from Google within weeks. To de-index a single page, use SEO Meta per page instead — not this site-wide switch.
Block AI crawlers
The Block AI crawlers checkbox adds explicit Disallow: / rules for the known AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and roughly ten more) to your robots.txt. Search engines are unaffected — only AI crawlers are blocked.
Blocking reduces visibility in AI chats
Leave this off for most sites. Blocking opts your content out of AI assistants (ChatGPT, Claude, Perplexity, …) — the opposite of being cited by them. Turn it on only if you deliberately do not want your content used by AI. Google-Extended controls only Gemini training, not your Google Search ranking.
llms.txt
llms.txt is a curated Markdown overview of your most important pages, served at /llms.txt. Some AI assistants read it to understand your site at a glance. It is not a sitemap (that is /sitemap.xml) — it is a short, human-curated map.
- Click Generate to build a suggestion from your published pages and blog posts. The editor fills with a Markdown draft — site name, a summary line, and a list of pages with descriptions.
- Edit the draft freely: trim it, reword the descriptions, add context.
- Click Save to publish it at
/llms.txt.
While the editor is empty, /llms.txt returns 404 — nothing is served until you generate (or write) content and save it.
Keep it curated
llms.txt works best as a short overview of your key pages, not an exhaustive URL dump. Generate, then prune it down to the pages that matter.
Saving
Save writes all three settings at once. Your robots.txt is rewritten immediately — you do not wait for the daily sitemap task. On a self-hosted install the change is live at once; behind a CDN it can take up to an hour to propagate at the edge.
Common issues
"Block AI crawlers" is not "Not indexed"
The two controls are independent. Block AI crawlers still lets Google index you; Not indexed hides you from everyone. Set each to what you actually want.
Removing the llms.txt
There is no separate delete button. To stop serving /llms.txt, clear the editor and click Save — an empty value makes the URL return 404.
noindex left on after staging
If you set Website indexing to Not indexed while working on a staging copy, switch it back to Indexed before launch. A forgotten noindex keeps the live site out of search.
See also
- SEO Meta per page — per-page title, description, and noindex.
- XML Sitemap — the machine-readable URL list (distinct from
llms.txt). - 301 Redirects — forward old URLs to new ones.