autoscout24.de — AI Site Grade

Analysis

No sitemap, no llms.txt, but a sophisticated AI-crawler gate

The single most non-obvious finding is that AutoScout24 operates a deliberate, tiered AI-crawler access policy — it explicitly blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot from the entire site root (Disallow: /) while carving out nine specific content paths they are allowed to crawl — yet every single AI bot tested still receives a full 200 response with ~900KB of content on the homepage, meaning the robots.txt rules are either not enforced at the server/WAF level or the site relies on crawler compliance rather than technical blocking.

Crawler Access

The robots.txt at autoscout24.de/robots.txt contains a sophisticated AI-bot blocklist: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot are all given Disallow: / followed by explicit Allow: directives for /informieren/, /auto/, /elektroauto/, /auto-verkaufen/, /fahrzeugbewertung/, /moto/, /motorrad-verkaufen/, /leasing/, and /finanzierung/. However, compare_bot_access on the homepage shows all 11 tested UAs (including GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider) return HTTP 200 with 900KB–1.2MB of content — no 403, no 404, no thin shell. The site runs on nginx behind AWS CloudFront with HSTS and X-Frame-Options: SAMEORIGIN, but no CSP or permissions-policy headers. The llms.txt returns HTTP 404 (a branded error page). The sitemap.xml and sitemap_index.xml both return HTTP 404 — no sitemap exists at any standard location.

Cold-Knowledge Gap

A frontier LLM queried cold about AutoScout24 knows the brand as a European multi-country marketplace founded in 1998, headquartered in Munich, with products like the mobile app, "Mein AutoScout24" alerts, and a "Fair Price" indicator. It also recalls a 2023 data-privacy criticism around user tracking. The site itself, however, makes no mention of the 1998 founding year, the Munich HQ, or any privacy controversies. The homepage and key pages position AutoScout24 as "Europas größter Online-Automarkt" (Europe's largest online car market) — a claim the cold model does not independently assert. The site also heavily promotes "smyle" (online car buying), "Direkt-Verkauf" (direct sale to dealers), and "SafeTrade" payment — products the cold model knows nothing about.

Schema Posture

The homepage carries only a bare Organization schema with name, legal name, URL, and social profiles. No WebSite, SearchAction, or WebPage schema on the homepage. Listing pages (e.g., /lst/audi, /lst/audi/a3) are stronger: they include BreadcrumbList, FAQPage (with 5–6 dynamically generated Q&A pairs about pricing, motorization, fuel consumption, colors, body types), Product with AggregateOffer (offer count, low/high price), and a Car type with manufacturer/model. The /elektroauto/modelluebersicht/ page also carries FAQPage schema. The /auto-verkaufen/ page has FAQPage schema with detailed Q&A about the selling process. No individual vehicle detail page could be fetched — the URL pattern /angebote/... redirects to the model listing page, suggesting individual listing pages may be JS-rendered or behind authentication.

External Signals

DNS records show the site uses AWS Route53 (ns-1447/161/1956/901.awsdns-*.org/com/co.uk/net), Microsoft 365 for mail (autoscout24-de.mail.protection.outlook.com), and verification TXT records for Google, Apple, and Facebook. The site serves assets from a CloudFront CDN (da209hnz4tkon.cloudfront.net). External links from the homepage point to autoscout24.com (careers) and autoscout24-media.de (media/ads). The footer links to LeasingMarkt.de as a sister property. No Reddit threads or press articles about AI visibility were found in search. The cold model's mention of a 2023 data-privacy criticism is not addressed or acknowledged anywhere on the site.

Findings

1. AI bot blocklist in robots.txt is not enforced at server level High

   Robots.txt explicitly disallows GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, and CCBot from the entire site root, then allows nine specific paths. However, all 11 tested AI bots receive a full 200 response with ~900KB of content on the homepage, indicating the rules are not enforced by the server or WAF.

   What to change: Enforce robots.txt rules at the server or WAF level to return 403 or 404 for disallowed AI bots, or remove the blocklist if all bots are welcome.

2. No sitemap.xml or sitemap_index.xml found High

   Both standard sitemap URLs return HTTP 404, and a web search for sitemap yielded no results. This severely limits crawler discovery of listing pages and other content.

   What to change: Generate and submit a sitemap covering all listing pages, category pages, and informational content.

3. No llms.txt file available Medium

   The llms.txt endpoint returns HTTP 404 with a branded error page, missing an opportunity to guide AI crawlers to key content.

   What to change: Create an llms.txt file that lists important pages like the homepage, listing categories, and informational sections.

4. Homepage lacks WebSite, SearchAction, and WebPage schema Medium

   The homepage only has a bare Organization schema. Missing WebSite schema with SearchAction reduces the chance of rich search results and AI understanding of site functionality.

   What to change: Add WebSite schema with SearchAction and WebPage schema to the homepage.

5. Cold LLM lacks knowledge of key products like smyle, Direkt-Verkauf, and SafeTrade Medium

   A frontier LLM queried cold about AutoScout24 did not mention smyle (online car buying), Direkt-Verkauf (direct sale to dealers), or SafeTrade payment, despite these being heavily promoted on the site. This indicates the site's content is not being effectively ingested by AI models.

   What to change: Ensure product pages are crawlable and include structured data (e.g., Product schema) to help AI models understand offerings.

6. Cold LLM knows founding year and HQ but site does not mention them Low

   The cold model correctly recalled AutoScout24 was founded in 1998 and headquartered in Munich, but the site itself does not prominently display this information, missing an opportunity to reinforce brand authority.

   What to change: Add a footer or about page that includes founding year and headquarters location.

7. 2023 data-privacy criticism not addressed on site Low

   The cold model recalled a 2023 data-privacy criticism around user tracking, but the site makes no mention of it or any corrective actions, which could erode trust with AI-generated summaries.

   What to change: Consider adding a privacy page or statement addressing past criticisms and current practices.

8. Individual vehicle detail pages may be JS-rendered or behind authentication High

   A sample listing URL redirected to the model listing page, suggesting that individual listing pages are not statically rendered or require authentication, making them invisible to crawlers.

   What to change: Ensure individual listing pages are server-side rendered and accessible to crawlers without authentication.

9. Missing CSP and permissions-policy headers Low

   The homepage lacks Content-Security-Policy and Permissions-Policy headers, which are not directly related to AI visibility but are best practices for security and could affect how AI tools render content.

   What to change: Add Content-Security-Policy and Permissions-Policy headers to improve security posture.

What's working

• FAQPage schema on listing and category pages — Listing pages like /lst/audi and /elektroauto/modelluebersicht/ include FAQPage schema with dynamically generated Q&A pairs about pricing, motorization, and fuel consumption, which helps AI models extract structured information.
• Product and Car schema with AggregateOffer on listing pages — Listing pages include Product schema with AggregateOffer (offer count, low/high price) and Car type with manufacturer/model, providing rich structured data for AI models.
• BreadcrumbList schema on listing pages — Listing pages include BreadcrumbList schema, aiding navigation understanding for crawlers and AI models.
• FAQPage schema on /auto-verkaufen/ page — The sell page includes FAQPage schema with detailed Q&A about the selling process, helping AI models understand the service.
• Robots.txt allows AI bots to crawl key content paths — Despite blocking the root, robots.txt explicitly allows AI bots to crawl /informieren/, /auto/, /elektroauto/, /auto-verkaufen/, and other important sections, ensuring some content remains accessible.
• Homepage serves full content to all bots — The homepage returns ~900KB of content to all tested AI bots, ensuring the brand and key messaging are visible.
• Organization schema on homepage with social profiles — The homepage includes Organization schema with name, legal name, URL, and social profiles, providing basic brand information to AI models.
• Informational content in /informieren/ section — The /informieren/ section contains test reports, news, and guides, providing valuable content that can be indexed by AI models.