RowanWeb Markdown for AI – .md Pages & llms.txt

AI assistants and RAG systems increasingly fetch site content directly, but they have to work through your theme’s HTML markup to do it — extra tokens, extra noise, and a real chance of misreading the actual content. This plugin gives them a clean alternative:

• {permalink}.md — a markdown version of every public page, generated from the same rendered content your visitors see (post_content, blocks, shortcodes) and converted with a real HTML → Markdown parser, not a regex stripper. Headings, tables, links and images keep their structure.

• /llms.txt — a dynamic index of public pages grouped by content type, with a short description for each. Every entry links both to the normal HTML page (so an AI citing your site points at a page a human can actually read) and to its.md version (for efficient re-fetching).

• /llms-full.txt (optional, generated on demand) — a single file with the full markdown content of your most recent posts per content type (50 per type by default; set any number, or no limit, on the Settings tab), for tools that want more than the short index. Generated with one click on the Diagnostics tab; after content changes, an admin notice reminds you to regenerate it.

• Deep custom-field support — if Advanced Custom Fields is active, the plugin automatically finds and renders every ACF field used on a page (any type, any nesting — repeaters, flexible content, groups) without any per-site configuration.

• Privacy-safe by default — password-protected posts, drafts, and anything flagged noindex by your SEO plugin are never served as markdown. Per-post exclusion is also available from the editor sidebar.

• Plays well with other SEO plugins — if Yoast SEO or Rank Math already generates its own llms.txt, this plugin automatically steps aside instead of racing for the same URL.

• Pre-warmed, not lazy — markdown is rebuilt the moment you save a post, so the first crawler to arrive never waits on a cold cache. A bulk warm-up button covers existing content after activation.

• robots.txt check — one click on the Diagnostics tab shows whether your robots.txt lets AI crawlers read the.md pages and llms files, which rule gets in the way and what to change. The plugin itself never edits robots.txt.

• Bot activity log — see which AI crawlers actually visit your.md / llms.txt URLs, how often, and which exact pages, right in the Activity tab.

• Multilingual-aware (Polylang) — on a multilingual site, llms.txt and llms-full.txt split into one section per language instead of mixing everything into a single list under one heading; custom-field labels are automatically translated to match each page’s own language.

A note on scope

), which your existing SEO plugin continues to handle exactly as before.

Pro version

RowanWeb Markdown for AI Pro is a separate, paid version. Everything described above works fully in this free plugin; Pro adds extra tools for busier sites:

• Cloudflare cache purge — extends your existing Cloudflare plugin’s cache purge (or purges directly via your own API token if none is active) to also cover.md pages, llms.txt, and llms-full.txt whenever a post is saved.

• Automatic llms-full.txt regeneration — the file rebuilds itself in the background shortly after relevant content is saved (bulk edits collapse into a single rebuild), so it never needs a manual click.

• Weekly AI-crawler list updates — picks up newly launched AI crawlers from ai.robots.txt every week instead of waiting for the next plugin release. A plain download of a public file (no site data is sent), and it can be switched off.

• CSV export — download the bot activity log as a CSV file for a spreadsheet or another analytics tool.

• Rate limiting — throttle a recognized bot making too many.md / llms.txt requests per minute from one IP.

• Suspicious-activity detection — flags requests probing for credentials or cloud metadata under a spoofed AI-bot User-Agent, listed separately from genuine crawler traffic.

• Extra Activity reports — filter “Recent hits” by URL, plus “Hits per day” and “Top pages” (most-requested URLs with a per-bot breakdown).

External services

This plugin connects to the following third-party services. Nothing else is sent anywhere.

DeepL API (optional custom-field label translation)

txt output.

• When: only after you add a DeepL API key — when you click “Translate field labels” on the Actions tab, or while a post is being saved / the cache is warmed / llms-full.txt is generated and a label hasn’t been translated yet. Never during a visitor’s or crawler’s request.

• What is sent: the field label texts, the target language (and the source language, if you set one), and your API key. Post content is never sent. Translations are cached, so each label is sent only once.

• Without an API key, nothing is ever sent to DeepL.

• DeepL Terms and Conditions and Privacy Policy.

Freemius (optional opt-in, licensing and updates of the Pro version)

The plugin includes the Freemius SDK, which handles the separate Pro version’s licensing and updates. In this free version, Freemius only shows an optional opt-in screen after activation.

• When: only if you choose to opt in (or activate a Pro license).

• What is sent: your site URL, WordPress and PHP versions, plugin version, site language, and the admin’s name and email address, plus plugin activation/deactivation/uninstall events. If you skip the opt-in, nothing is sent.

• Freemius Terms of Service, Privacy Policy and details on what is collected.

Support

ru/en/blog/rowanweb-markdown-for-ai-plugin-for-ai-crawlers/