CrawlBrief for LLMs.txt

txt file and optional Markdown versions of public WordPress content for AI assistants, AI agents and large language models.

txt, rel="alternate" type="text/markdown" links for eligible WordPress content, and matching HTTP Link metadata on Markdown responses.

CrawlBrief combines automatic generation with editorial control. Instead of publishing an uncontrolled list of every URL, you can choose the pages that matter, organize important resources, edit descriptions, feature or exclude content, and provide clean Markdown representations that compatible AI tools can retrieve when needed.

WordPress llms.txt Generator

CrawlBrief automatically publishes:

• /llms.txt – a structured, LLM-friendly overview of your WordPress website.

• /index.html.md – a Markdown representation of the homepage.

• /example-post/index.html.md – clean Markdown versions of eligible posts, pages and supported public post types.

The endpoints are served dynamically through WordPress. txt or Markdown files on your server.

llms.txt v2 Support

txt v2 proposal:

• Adds rel="describedby" so public pages can identify the applicable /llms.txt file.

• Adds rel="alternate" type="text/markdown" for pages with a Markdown representation.

• Sends matching discovery information through HTTP Link headers on Markdown responses.

• Keeps the v2-compatible index.html.md URL format for WordPress URLs without filenames.

• Links directly to clean Markdown resources from generated llms.txt sections when available.

Automatic and Manual Page Selection

txt file:

• Automatic – detect important WordPress pages automatically.

• Automatic + Custom – use automatic detection and add your own selected WordPress resources.

• Manual – control the selected resources yourself.

CrawlBrief can detect common site pages such as:

• Homepage

• About page

• Contact page

• Privacy Policy

• Terms page

• WordPress posts page

• WooCommerce shop page when available

Automatically detected resources can be excluded individually.

Searchable WordPress Resource Selection

Add important WordPress pages using searchable selectors instead of manually copying URLs.

txt entry.

You can create separate selections for:

• Start Here resources

• Optional resources

Additional external URLs, archives and documentation hosted outside WordPress can still be added through the Advanced additional-links section.

Editable llms.txt Descriptions

CrawlBrief lets you control how important resources are described to AI systems.

You can:

• Override descriptions for automatically detected pages.

• Add description overrides to selected Start Here resources.

• Add description overrides to selected Optional resources.

• Set a dedicated llms.txt description on individual posts or pages.

If no override is supplied, CrawlBrief can fall back to the available WordPress description, excerpt, content-derived text or built-in role description.

Editorial Control

Each supported WordPress resource can use CrawlBrief’s editorial controls:

• Automatic – let CrawlBrief include the resource normally.

• Featured – prioritize the resource in a dedicated Featured Resources section.

• Excluded – prevent the resource from appearing in generated CrawlBrief sections.

txt files rather than simple URL dumps.

Structured llms.txt Sections

Depending on your settings and content, CrawlBrief can generate sections for:

• Start Here

• Featured Resources

• WordPress categories

• Public post types

• Pages

• Site Indexes

• Optional resources

You can configure the number of links per section, choose supported public post types, exclude selected category slugs, include the WordPress sitemap, and control optional sections.

CrawlBrief also prevents curated Optional resources from being duplicated in general page sections.

Clean Markdown for AI Assistants

Optional Markdown representations remove much of the surrounding theme and navigation markup and expose useful page content in a simpler text format.

Markdown output can contain:

• Page or post title

• Short description

• Canonical URL

• Author

• Last-updated date

• Featured image

• Headings

• Paragraphs

• Lists

• Links

• Main page content

If a static WordPress homepage has no stored page body, CrawlBrief can use the configured site summary and detailed context so the homepage Markdown endpoint is still useful.

Caching and Automatic Regeneration

CrawlBrief includes an optional generated-output cache to reduce repeated database work.

It can:

• Cache generated llms.txt output.

• Regenerate cached content on a daily schedule.

• Clear generated output when relevant WordPress content changes.

• Invalidate stale output when CrawlBrief is upgraded.

• Clear cached output when settings are saved.

• Integrate with common WordPress caching plugins when available.

Privacy and Lightweight Operation

CrawlBrief performs generation inside your WordPress installation.

• No external API is required.

• No account is required.

• No visitor analytics are collected by CrawlBrief.

• Site content is not automatically sent to LLMs.txt Lab or another third-party service.

• No public footer credit or backlink is added.

• No physical llms.txt or Markdown files need to be written to the server.

Who Is CrawlBrief For?

txt convention.

It can be used on:

• Blogs

• Educational websites

• Documentation sites

• Business websites

• News and publishing sites

• WooCommerce stores

• Knowledge bases

• Membership and learning websites

• Sites using public custom post types

What Is llms.txt?

txt is an emerging Markdown-based convention for giving AI systems a concise overview of a website and links to important machine-readable resources.

txt and XML sitemaps. CrawlBrief focuses on helping compatible AI assistants and agents discover and retrieve useful site content; it does not promise higher Google rankings or guaranteed inclusion in AI-generated answers.

Works With AI Assistants and Agents

CrawlBrief produces standard web and Markdown resources that can be accessed by compatible AI assistants, agents and retrieval tools, including tools associated with ecosystems such as ChatGPT, Claude, Perplexity and other large language model applications.

txt depends on the individual AI system or crawler.

Optional LLMs.txt Lab Tools

txt inside WordPress. txt Lab provides optional browser-based tools:

• Validate llms.txt – check an existing llms.txt file or URL.

• Generate llms.txt – manually create an llms.txt file when needed.

• LLMs.txt Lab – access additional llms.txt tools and resources.

These tools are optional. txt Lab.

How to Verify the Plugin

txt. The settings page shows verification URLs for your website.

txt file, visit:

txt

To check the homepage Markdown endpoint, visit:

md

To check a post or page Markdown endpoint, use the existing CrawlBrief pattern:

md

txt URL should load as plain text. A post or page Markdown URL should load clean Markdown content with the title, canonical URL, last-updated date, author when available, and page body. The homepage Markdown endpoint contains the available homepage or configured site information.

txt v2 discovery, view the source of a public HTML page. CrawlBrief outputs:

txt">

For an eligible post or page with Markdown representations enabled, it also outputs:

md">

txt file and canonical HTML page through the HTTP Link header.

LLMs.txt v2 Compatibility

txt v2 discovery model while retaining backward compatibility with existing CrawlBrief installations.

• /llms.txt remains at the site root and covers the site’s public URLs.

• Existing /index.html.md and /post/index.html.md representations are retained. llms.txt v2 explicitly permits index.html.md for URLs without file names.

• Agents can discover Markdown representations through rel=”alternate” type=”text/markdown” rather than guessing URL transformations.

• Public HTML pages point to the covering llms.txt with rel=”describedby”.

• Markdown responses expose the same relationship through an HTTP Link header.

• The Optional section remains available as a convention for secondary resources that agents may skip when a shorter context is needed; it is not treated as a special processing instruction.

• Category and custom-post-type archive links remain normal HTML links unless CrawlBrief can actually serve a Markdown representation for the URL.

txt proposal remains an evolving community specification. org for the current proposal.

Cache Compatibility

The plugin clears its internal transient cache when configured content changes. It also attempts to purge common caching plugins when available, including LiteSpeed Cache, WP Rocket, W3 Total Cache, WP Super Cache and SiteGround Optimizer.

Other caching plugins can hook into:

crwbrf_txt_cache_purged

Privacy

txt content or site data to a third-party service. All generation happens inside WordPress using public site content and settings saved by the site administrator. txt Lab, are opened only when an administrator chooses to visit them.

txt data through those links.