Not sure where to start? Take a guided tour.
AI Site Grader
AI Query Scanner
llms.txt GeneratorQuery transformation severity tracker
The benchmark data indicates a consistent pattern where AI models actively restructure user prompts rather than passing them directly to search systems. Exact matches are statistically negligible at 0.17 percent, while expanded queries account for 55.5 percent of the dataset, demonstrating a tendency to lengthen and constrain search parameters. Complete rewrites occur in 31.85 percent of cases, resulting in an average Jaccard similarity of 25.24 percent between the original prompt and the final query.
Query transformation severity tracker for the study How AI Translates Your Questions.
The key numbers.
- Exact match rate
- Only 20 of 11,521 pairs matched exactly.
- 0.17%
- Average similarity
- Average Jaccard similarity between prompt and search query.
- 25.24%
- Complete rewrites
- 3,670 pairs fell into the complete rewrite bucket.
- 31.85%
- Expanded queries
- AI made the query longer in 6,392 cases.
- 55.5%
The ranked view.
- 155.5%
Expanded queries
More than half of queries grow longer and more constrained.
- 231.85%
Complete rewrites
Nearly one third of pairs keep very little original phrasing.
- 325.24%
Average similarity
Prompt and search query often overlap far less than teams assume.
- 40.17%
Exact matches
Literal user phrasing is almost never the final retrieval query.
What changed
Expanded queries reach 55.5 percent
More than half of queries grow longer and more constrained, with the AI making the query longer in 6,392 cases.
Complete rewrites account for 31.85 percent
Nearly one third of pairs keep very little original phrasing, as 3,670 pairs fell into the complete rewrite bucket.
Average similarity measures 25.24 percent
Prompt and search query often overlap far less than teams assume, reflecting the average Jaccard similarity between prompt and search query.
Exact matches fall to 0.17 percent
Literal user phrasing is almost never the final retrieval query, with only 20 of 11,521 pairs matching exactly.
| Measure | Result |
|---|---|
| Exact match rate Only 20 of 11,521 pairs matched exactly. | 0.17% |
| Average similarity Average Jaccard similarity between prompt and search query. | 25.24% |
| Complete rewrites 3,670 pairs fell into the complete rewrite bucket. | 31.85% |
| Expanded queries AI made the query longer in 6,392 cases. | 55.5% |
Population and method
Built from 11,521 captured prompt-to-query pairs observed in OpenAI web search calls, with 100% search-query coverage in the sampled dataset.
Where the evidence stops
This study reflects captured search-query transformations rather than every hidden retrieval step a model may perform.
Some inserted entities reflect category heuristics or market leaders rather than final recommendation outcomes.
The underlying dataset is concentrated in commercial and research prompts rather than all possible conversational traffic.
Cite this tracker.
Source study: Query Translation“The benchmark data indicates a consistent pattern where AI models actively restructure user prompts rather than passing them directly to search systems. Exact matches are statistically negligible at 0.17 percent, while expanded queries account for 55.5 percent of the dataset, demonstrating a tendency to lengthen and constrain search parameters. Complete rewrites occur in 31.85 percent of cases, resulting in an average Jaccard similarity of 25.24 percent between the original prompt and the final query.”
Mack Grenfell · Query transformation severity tracker
How AI Translates Your Questions · Trakkr Research · Updated 22 January 2026