What is AIWebIndex? AI crawler guide
AIWebIndex is a crawler or fetcher associated with Lyrenth. See source-graded user-agent, robots.txt, verification, and observation guidance.
What is AIWebIndex?
AIWebIndex is a documented Lyrenth crawler or fetcher. Crawler that builds Lyrenth, an index of public web pages served to AI systems and developers as extracted, attributed documents.
Evidence status
| Field | Value |
|---|---|
| Evidence | Officially documented |
| Lifecycle | Active |
| Purpose | search |
| robots.txt posture | honors |
| Source checked | 2026-09-17 |
Documented user-agent
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
robots.txt allow example
User-agent: AIWebIndex Allow: /
robots.txt block example
User-agent: AIWebIndex Disallow: /
What the rule can and cannot do
Robots.txt expresses an access policy to compliant automated crawlers. It does not authenticate the sender, remove content already collected, or guarantee that a model will use or not use content.
How to verify a request
- Match the complete documented user-agent where one exists. A match is only a clue.
- Use the operator-published check: Published IP list.
- Keep the observed request, operator documentation, and any inference as separate fields.
JavaScript behavior
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
Related bots
- LinkupBot: Another search index built for AI applications.
- ExaSearchBot: Another search index that AI applications and agents query.
- ShapBot: Also tracked as a search crawler.
- FirecrawlAgent: Another service that turns web pages into content for AI applications.
- Kagibot: Also tracked as a search crawler.
- MistralAI-Index: Also tracked as a search crawler.
- OAI-SearchBot: Also tracked as a search crawler.
- Robots.txt: Robots.txt is the control file used to allow or block AIWebIndex.
- AI training opt-out: Lyrenth says AIWebIndex is not used to train foundation models.
Frequently Asked Questions
What is AIWebIndex?
AIWebIndex is a officially documented crawler or fetcher record associated with Lyrenth.
What user-agent does AIWebIndex use?
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index). Check the evidence status before attributing a matching request.
Can AIWebIndex be blocked in robots.txt?
The record identifies AIWebIndex as the token to review. Robots.txt is a request policy, not proof of model use or non-use.
How can I verify a AIWebIndex request?
Start with the full user-agent, then use published ip list from the operator. A user-agent match alone is not proof.
Data & Sources
- Lyrenth documentation - Primary source for AIWebIndex crawler details.
- AIWebIndex source reference - Source used to verify AIWebIndex.
- AIWebIndex verification: Published IP list - Lyrenth also signs requests with Web Bot Auth (RFC 9421) and says every crawler IP has forward-confirmed reverse DNS under lyrenth.com.