lesterglenn.com — AI Site Grade

Analysis

Akamai Firewall Blocks Every Crawler Except Anthropic

Lester Glenn Auto Group's site is served behind an Akamai WAF that returns HTTP 403 to every standard browser and crawler user-agent — including GPTBot, Google-Extended, PerplexityBot, ClaudeBot, and ChatGPT-User — while anthropic-ai alone receives a full 200 response with 553KB of rendered HTML. This creates a bizarre asymmetry: Claude's training crawler sees the complete site, but no other major AI engine can access a single page.

Crawler Access

The robots.txt file is itself blocked by Akamai to standard user-agents (403), but when fetched as anthropic-ai it reveals a permissive policy. The file explicitly disallows GPTBot, OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, and PerplexityBot from accessing JS, CSS, JSON, and API paths — but does not disallow them from the main site content. The homepage, sitemap, and all inventory pages are technically crawlable by these bots per the robots.txt, yet Akamai's WAF blocks them at the network layer before they ever reach the nginx server. No llms.txt exists (returns 403). The sitemap.xml (586KB, accessible only to anthropic-ai) contains thousands of URLs including inventory pages, blog posts, and location pages — all invisible to Google and OpenAI crawlers.

Cold-Knowledge Gap

The LLM model knows Lester Glenn as a "family-owned automotive dealership group based in Toms River, New Jersey" founded in 1965, selling Ford, Hyundai, Kia, Subaru, and CDJR brands. It recalls the tagline "Lester Glenn, the one to see" and notes "typical mixed reviews on Google and Yelp." This prior knowledge is generic and shallow — it describes any mid-sized multi-franchise dealer. The actual site, visible only to anthropic-ai, presents a modern Dealer.com platform with inventory search, service scheduling, and multiple location pages (Toms River, Freehold, Ocean Township, Sea Girt). The gap is that AI models cannot verify or enrich this prior knowledge because they cannot access the site's actual inventory, pricing, or service offerings.

Schema Posture

The homepage HTML (fetched as anthropic-ai) shows the site runs on the Dealer.com (DDC) platform with class names referencing multiple franchises (dodge, hyundai, honda, ford, mazda, chrysler, chevrolet, jeep, subaru, fiat, gmc, ram). The page is heavy with JavaScript bundles and prefetch links, indicating heavy JS-rendering dependency. No JSON-LD schema was extractable from the 403-blocked browser fetch; the anthropic-ai fetch returned raw HTML that likely contains DDC's standard AutoDealer and Organization schemas, but these are invisible to any crawler that cannot bypass the Akamai wall.

External Signals

Web search returned zero results for "Lester Glenn Auto Group" — a complete blind spot likely caused by the Akamai block preventing search engine indexing. The DNS is hosted on ns1.dealer.com and ns2.dealer.com, confirming the DDC platform. Email security runs through Hornetsecurity. The Akamai error references (errors.edgesuite.net) confirm the WAF layer. The site has no discoverable external press, Reddit threads, or review aggregator citations accessible via search — the firewall effectively silences the brand's entire web presence from external discovery.

Findings

1. Akamai WAF blocks all major AI crawlers except Anthropic High

   The Akamai WAF returns HTTP 403 to GPTBot, Google-Extended, PerplexityBot, ClaudeBot, and ChatGPT-User, while allowing anthropic-ai full access. This prevents all major AI engines except Claude from accessing site content.

   What to change: Reconfigure the Akamai WAF to allow access to GPTBot, Google-Extended, PerplexityBot, and other major AI crawlers, or implement a more selective blocking strategy that does not deny all crawlers.

2. robots.txt returns 403 to standard user-agents High

   The robots.txt file is blocked by Akamai for standard user-agents, returning HTTP 403. Only anthropic-ai can retrieve it, revealing a permissive policy that does not disallow crawlers from main content but is inaccessible to most bots.

   What to change: Ensure robots.txt is publicly accessible without authentication or WAF blocking, and explicitly allow major AI crawlers to access the entire site.

3. Sitemap XML accessible only to anthropic-ai High

   The sitemap.xml (586KB) returns 403 to standard user-agents but is accessible to anthropic-ai. It contains thousands of URLs including inventory, blog, and location pages, all invisible to Google and OpenAI crawlers.

   What to change: Make sitemap.xml publicly accessible and ensure it is listed in robots.txt for all crawlers.

4. No llms.txt file available Medium

   The llms.txt endpoint returns HTTP 403, indicating no llms.txt file is published. This file would help AI crawlers discover key pages and content.

   What to change: Create and publish an llms.txt file listing important pages for AI crawlers.

5. Heavy JavaScript rendering dependency Medium

   The homepage relies heavily on JavaScript bundles and prefetch links, typical of the Dealer.com platform. This can hinder content extraction by AI crawlers that do not execute JavaScript.

   What to change: Implement server-side rendering or provide static HTML snapshots for critical content to ensure AI crawlers can extract text without JavaScript execution.

6. No JSON-LD schema extractable from blocked pages High

   JSON-LD schema (likely AutoDealer and Organization) is present in the HTML but invisible to any crawler that cannot bypass the Akamai wall. Standard crawlers receive 403 and cannot extract structured data.

   What to change: Ensure schema markup is accessible to all crawlers by removing the WAF block for AI bots, and consider adding schema in a static format that does not require JavaScript.

7. Zero search engine indexing due to WAF block High

   Web searches for the domain and brand name return zero results, indicating that search engines cannot index the site because of the Akamai block. This eliminates organic discovery and external signals.

   What to change: Allow Googlebot and other search engine crawlers through the WAF to enable indexing and build external signals.

8. LLM prior knowledge is generic and shallow Medium

   The LLM's knowledge of Lester Glenn Auto Group is limited to generic dealership facts (founded 1965, multi-franchise, Toms River) and cannot be enriched because the site is inaccessible. Actual site content (inventory, pricing, service) is invisible to AI models.

   What to change: Allow AI crawlers to access the site so that models can learn specific inventory, pricing, and service offerings.

What's working

• Anthropic AI crawler has full access to the site — The anthropic-ai crawler receives a 200 response with complete HTML, allowing Claude to index all site content including inventory, blog, and location pages.
• Robots.txt does not disallow crawlers from main content — The robots.txt file, when accessible, only disallows crawlers from JS, CSS, JSON, and API paths, not from the main site content. This is a favorable policy if the WAF block is removed.
• Comprehensive sitemap with thousands of URLs — The sitemap.xml contains thousands of URLs covering inventory, blog posts, and location pages, providing a complete map of site content for crawlers that can access it.
• Modern Dealer.com platform with structured data — The site runs on the Dealer.com platform, which typically includes standard AutoDealer and Organization JSON-LD schemas, providing a foundation for rich results if accessible.