What is Cotoyogi? AI crawler guide
Cotoyogi is a crawler or fetcher associated with ROIS-DS. See source-graded user-agent, robots.txt, verification, and observation guidance.
What is Cotoyogi?
Cotoyogi is a documented ROIS-DS crawler or fetcher. Crawler run by the Center for Research and Development on Data Lake at ROIS-DS in Japan to collect Japanese language data resources.
Evidence status
| Field | Value |
|---|---|
| Evidence | Officially documented |
| Lifecycle | Active |
| Purpose | other |
| robots.txt posture | honors |
| Source checked | 2026-09-17 |
Documented user-agent
Mozilla/5.0 (compatible; Cotoyogi/4.0; +https://ds.rois.ac.jp/center8/crawler/)
robots.txt allow example
User-agent: Cotoyogi Allow: /
robots.txt block example
User-agent: Cotoyogi Disallow: /
What the rule can and cannot do
Robots.txt expresses an access policy to compliant automated crawlers. It does not authenticate the sender, remove content already collected, or guarantee that a model will use or not use content.
How to verify a request
- Match the complete documented user-agent where one exists. A match is only a clue.
- Use the operator-published check: Published IP range.
- Keep the observed request, operator documentation, and any inference as separate fields.
JavaScript behavior
The cited operator material does not verify JavaScript rendering. Serve useful HTML before client-side JavaScript where possible.
Related bots
- SBIntuitionsBot: Another crawler from a Japanese operator.
- CCBot: Another crawler that collects public web data for research datasets.
- AI2Bot: Another crawler run by a research organization.
- ICC-Crawler: Another crawler run by a Japanese research organization.
- Panscient: Also tracked as a general crawler.
- Diffbot: Also tracked as a general crawler.
- omgili: Also tracked as a general crawler.
- Robots.txt: Robots.txt is the control file used to allow, slow, or block Cotoyogi.
- Training data: Public AI crawler lists often group Cotoyogi with data collection crawlers.
Frequently Asked Questions
What is Cotoyogi?
Cotoyogi is a officially documented crawler or fetcher record associated with ROIS-DS.
What user-agent does Cotoyogi use?
Mozilla/5.0 (compatible; Cotoyogi/4.0; +https://ds.rois.ac.jp/center8/crawler/). Check the evidence status before attributing a matching request.
Can Cotoyogi be blocked in robots.txt?
The record identifies Cotoyogi as the token to review. Robots.txt is a request policy, not proof of model use or non-use.
How can I verify a Cotoyogi request?
Start with the full user-agent, then use published ip range from the operator. A user-agent match alone is not proof.
Data & Sources
- ROIS-DS documentation - Primary source for Cotoyogi crawler details.
- Cotoyogi verification: Published IP range - ROIS-DS lists 157.1.136.4 to 157.1.136.11 as the crawler's IP range.