Do AI crawlers prefer Markdown?
Every few weeks someone posts with total confidence that AI crawlers love Markdown, or that they ignore it entirely. Nobody seems to run the test. So we did: same URL, two formats, five crawlers. They don't agree with each other.
Half the GEO industry will tell you AI crawlers love Markdown. The other half says it makes no difference. Almost nobody has the data to back either claim. We built the test that people keep talking about but nobody seems to actually run: every public page on trakkr.ai gets randomly served as either Markdown or HTML, that assignment stays locked to the URL, and we track what five different AI crawlers do with each version.
Short version: GPTBot is the only crawler with a clear format preference, and it leans hard toward HTML. ClaudeBot's earlier HTML skew has faded to parity. OAI-SearchBot, ChatGPT-User, and PerplexityBot still reach Markdown and HTML at roughly the same rate. So Markdown is not winning more search or live retrieval coverage, but it is not losing it either.
What I find more interesting is actually underneath the headline results. Once you strip the nav, scripts, and tracking pixels out of a typical HTML page, what's left is just the answer. The Markdown version is about a fifth of the size. Cheaper to fetch, faster to parse. If retrieval cost ever becomes a routing signal inside the AI labs (and I'd bet it already is in at least one of them), that's the thing to watch. The numbers below use the latest reviewed snapshot.
The experiment
Same page. Two formats. Locked to the URL.
Every public page on trakkr.ai gets one of two surfaces (Markdown or HTML), based on a hash of its URL. Same content, same canonical, different format. The crawlers don't know they're in a test; they just see a page. And because the assignment is locked to the URL, the same page always serves the same surface. When a crawler comes back, we know which arm it's on. That's what makes the comparison clean.
Every public trakkr.ai URL enters once, then stays put for the rest of the experiment.
Concentrated across 0 recommendation pages, our most heavily crawled URLs.
Every public, indexable trakkr.ai page with a stable URL. Login-walled routes, redirects, and preview pages are left out.
Each URL is hashed with the experiment ID, and the hash decides Markdown or HTML. Same URL, same surface, every fetch; a crawler can't end up seeing both versions. Observed split: 50.0% Markdown, the rest HTML.
Our pages at /ai-recommends/<product>/<audience> (think “best AI transcription for nonprofits”). There are thousands of them, which is why this slice has enough volume to detect small effects. The results scoreboard runs against this slice.
Three OpenAI bots, three different jobs
OpenAI runs three crawlers. Most write-ups blur them into one.
They shouldn't. Each one does a different job, runs on a different schedule, and responds to format differently. If you count GPTBot fetches as evidence of live citations, or read ChatGPT-User numbers as proof of search indexing, you're measuring the wrong thing. We keep all three separate from here.
OAI-SearchBot
Search indexing crawler
Whether a page is being fetched for OpenAI's search index and citation-surfacing systems. Closest OpenAI signal for future ChatGPT Search discoverability.
ChatGPT-User
Live retrieval fetcher
Whether ChatGPT opened a public page during a user-triggered retrieval action. A conversation-time demand signal, not an automatic search crawler.
GPTBot
Training crawler
Whether OpenAI's training crawler requested the page. Useful for crawl behavior, but not evidence that the page was retrieved for a live citation.
The results so far
One clear outlier. Four neutral crawlers.
On our recommendation pages, GPTBot is the only one of the 5 crawlers with a clear Markdown-vs-HTML preference, and it leans hard toward HTML. The other 4 sit flat, reaching both formats at about the same rate. Click any row to see the numbers behind it.
ChatGPT-User is directionally HTML-skewed on best-for pages.
The search crawler is slightly more likely to cover Markdown-assigned best-for pages so far, but the difference is not strong enough to call.
PerplexityBot is directionally HTML-skewed on best-for pages.
ClaudeBot is directionally HTML-skewed on best-for pages.
GPTBot is behaving unlike the search crawler; the study reports it as a training-crawler anomaly rather than a live citation signal.
How to read this
GPTBot is the one settled result. On recommendation pages it reached 67.1% of the HTML arm and just 5.0% of the Markdown arm. ClaudeBot no longer shows the milder HTML skew we saw in May: it has now reached almost every page in both arms. GPTBot is a training crawler, not the system deciding whether your page gets cited in a live ChatGPT answer.
OAI-SearchBot is the result worth watching because it supports ChatGPT Search. Its Markdown edge on recommendation pages is now just 0.5 percentage points, with p=0.742. That is noise, not evidence of a preference. The snapshot history below shows how the early lead flattened as coverage filled in.
The crawlers that actually shape live answers (ChatGPT-User and PerplexityBot) sit flat. Markdown and HTML get reached at about the same rate. Which makes sense if you think about it: at conversation time these systems are chasing the user's question, not the page's format. That flat line is the real result here. The only bot avoiding Markdown is the training crawler; the bots involved in current search and live retrieval do not show a format preference.
What people actually ask about
Live retrieval follows the question, not the format.
ChatGPT-User isn't a crawler on a schedule. It opens a page mid-conversation because someone asked ChatGPT something and the model needed a real page to answer. Across 2.7M of these live fetches on our site, demand tracks the topics people are actually asking about, and it spreads roughly evenly across both arms of the experiment. Format doesn't seem to matter here. The question does.
Sanity check: ChatGPT-User reached 0.0% of Markdown-assigned pages and 0.0% of HTML-assigned pages. That's within 0.0% of each other (p=1.000). So the category gaps above are about what people asked, not which arm of the experiment got more traffic.
What Markdown changes today
smaller than the same page in HTML
OAI-SearchBot is effectively neutral, so the case for Markdown is not extra crawler reach. It is efficiency. Strip the nav, scripts, tracking pixels, and CSS chrome from a typical HTML page and what's left is just the answer. The Markdown version of the same page is about a quarter of the size. It is cheaper to fetch and faster to parse, without a measurable coverage penalty among search and live retrieval bots.
What changed over time
The early Markdown lead flattened out.
Every point below is a reviewed snapshot of the same randomized experiment. The line we watched most closely was OAI-SearchBot's Markdown edge on recommendation pages. It has narrowed to 0.5 percentage points and remains statistically neutral. The experiment keeps collecting events through August 18.
Reviewed snapshot.
How this study works
How this is built, and what it can’t yet say.
Every Monday's run produces one snapshot that the whole page is built from. If the prose ever says something the data doesn't back, the snapshot is what's true. Below is the short version of how it works, what it can't tell you yet, and any caveats from this week.
How it’s built
04- 01
Eligible pages are assigned to Markdown or HTML with a deterministic hash of experiment ID and canonical path.
- 02
The primary endpoint is page-level coverage: the share of assigned pages each bot reached at least once.
- 03
Request counts, revisits, and transferred bytes are secondary outcomes because a small number of popular URLs can dominate raw volume; variant comparisons use two-proportion z-tests and are reported as percentage-point differences with p-values.
- 04
ChatGPT-User is separated from OAI-SearchBot because it represents user-triggered live retrieval rather than search indexing.
What it can’t say
04- 01
The baseline event table currently reports all bot identity as user-agent-only rather than Cloudflare verified-bot enriched.
- 02
The first window did not serve ChatGPT-User as its own experiment cohort; it is shown as historical live-retrieval demand until the public launch.
- 03
The experiment measures crawl and retrieval behavior, not whether a downstream AI answer ultimately cited the page.
- 04
Legacy recommendation URLs are aggregated as public retrieval demand and are not presented as active experiment treatment pages.
- C01
Bot identity in this baseline window is user-agent based; Cloudflare verified-bot enrichment has not yet appeared in the experiment rows.
- C02
ChatGPT-User was measured as historical live-retrieval demand in the baseline and becomes a first-class interaction cohort from 2026-05-06T09:00:00Z.
- C03
Coverage is page-level and randomized by canonical path, so raw request volume is reported as a secondary signal.
See how your brand performs in AI search