fredbeans.com — AI Site Grade

Analysis

Fred Beans Auto Group's production website is entirely inaccessible to every major AI crawler and browser alike, yet the anthropic-ai bot receives a full 404KB HTML response — a bizarre inversion where only Anthropic's training crawler sees the site's actual content.

Crawler Access

The site is hosted behind Akamai (edgesuite.net) on the www subdomain, while the bare domain (fredbeans.com) refuses connections entirely. Every tested bot — GPTBot, Google-Extended, ClaudeBot, PerplexityBot, OAI-SearchBot, ChatGPT-User, and standard browsers — receives a 403 Access Denied from Akamai. The robots.txt is also blocked by Akamai for browser UAs. The sole exception is anthropic-ai, which bypasses Akamai and hits an nginx origin server with a 200 response. The robots.txt served to anthropic-ai contains rules for GPTBot, OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, and PerplexityBot — all disallowed from /api/, /apis/, /pixall/, and static asset paths — but no User-agent: * catch-all, meaning the default is to allow crawling. The llms.txt exists and is a massive 2.6MB file listing hundreds of pages with descriptions, making it one of the largest llms.txt files observed.

Cold-Knowledge Gap

The LLM model describes Fred Beans as a family-owned group founded in 1976, selling Ford, Chevrolet, Toyota, Honda, Hyundai, and Subaru, with a "Fred Beans Difference" no-haggle campaign. The actual site reveals a much larger operation: the homepage HTML class list includes 18+ brands (Kia, Hyundai, Buick, Acura, Ford, Mazda, Genesis, Jeep, Volkswagen, Ram, Dodge, Honda, Cadillac, Chrysler, Chevrolet, Lincoln, Subaru, Toyota, Nissan, Isuzu, GMC). The model's knowledge is missing at least half the brand portfolio and underestimates the group's scale.

Schema Posture

The homepage and inventory pages contain no JSON-LD schema of any kind. The schema extraction returned empty arrays for all fetched pages. Given the site runs on the Dealer.com (DDC) platform — a major automotive CMS — the absence of AutoDealer, Organization, or Product schema is a significant omission. The llms.txt compensates partially by providing structured page descriptions, but no machine-readable entity definitions exist.

External Signals

The site's Akamai WAF blocks all search engine bots and browsers, meaning Google cannot index the site's content. No external search results were returned for fredbeans.com queries. The Wayback Machine shows the site was accessible as recently as November 2024, suggesting the Akamai block is a recent configuration change or misconfiguration. The site uses Dealer.com's platform (ddc-site class), which typically generates JSON-LD automatically — the absence suggests a platform configuration issue or a template override that stripped schema output.

Findings

1. All major AI crawlers and browsers blocked by Akamai WAF High

   The site returns 403 Access Denied for GPTBot, Google-Extended, ClaudeBot, PerplexityBot, OAI-SearchBot, ChatGPT-User, and standard browsers. Only anthropic-ai receives a 200 response from the origin server.

   What to change: Reconfigure Akamai WAF to allow Googlebot, GPTBot, ClaudeBot, and other major AI crawlers to access the site, or serve a static version to bots.

2. Bare domain fredbeans.com refuses all connections High

   The bare domain fredbeans.com does not respond to HTTP requests, returning connection refused for all tools. The site only resolves on www.fredbeans.com.

   What to change: Configure DNS and web server to redirect fredbeans.com to www.fredbeans.com or serve content directly.

3. No JSON-LD schema on homepage or inventory pages High

   The homepage and new inventory page contain no JSON-LD schema for AutoDealer, Organization, or Product. The site runs on Dealer.com platform which typically generates schema automatically, suggesting a configuration issue.

   What to change: Enable JSON-LD schema generation in the Dealer.com platform for AutoDealer, Organization, and VehicleOffer types.

4. robots.txt inaccessible to browsers and most bots High

   The robots.txt file returns 403 for browser user agents, preventing search engines from discovering crawl rules. Only anthropic-ai can access it.

   What to change: Allow public access to robots.txt by configuring Akamai to serve it without authentication.

5. Sitemap inaccessible to browsers and most bots High

   The sitemap.xml is only accessible to anthropic-ai; browsers and other bots receive connection errors or blocks.

   What to change: Make sitemap.xml publicly accessible to all crawlers.

6. llms.txt file is 2.6MB, potentially overwhelming for LLMs Medium

   The llms.txt file is 2.6MB, listing hundreds of pages with descriptions. While comprehensive, its size may exceed context limits or cause LLMs to truncate or ignore it.

   What to change: Reduce llms.txt to a curated list of key pages (e.g., top 50) and use llms-full.txt for the complete listing.

7. LLM knowledge of Fred Beans is outdated and incomplete Medium

   The LLM model describes Fred Beans as selling 6 brands, but the site lists 18+ brands. The model's knowledge underestimates the group's scale and brand portfolio.

   What to change: Ensure the site is indexed by Google and other search engines to update the LLM's training data; publish a knowledge graph or structured data.

8. No external search results for fredbeans.com High

   Web searches for fredbeans.com and related queries returned zero results, indicating the site is not indexed by Google or Bing.

   What to change: Remove the Akamai block for Googlebot and Bingbot to allow indexing.

9. Akamai block appears to be a recent misconfiguration Medium

   Wayback Machine snapshots show the site was accessible in November 2024 and December 2022, suggesting the current WAF block is a recent change or misconfiguration.

   What to change: Review Akamai WAF rules to restore public access.

What's working

• anthropic-ai bot is whitelisted and receives full content — The anthropic-ai bot bypasses Akamai and receives a 200 response with full HTML content (404KB), including homepage and inventory pages.
• llms.txt file published with extensive page descriptions — The site publishes a 2.6MB llms.txt file listing hundreds of pages with descriptions, providing structured content for LLMs that can access it.
• robots.txt served to anthropic-ai allows crawling by default — The robots.txt served to anthropic-ai has no User-agent: * rule, meaning crawling is allowed by default, with only specific paths disallowed for certain bots.
• Sitemap accessible to anthropic-ai with 1.6MB of URLs — The sitemap.xml is served to anthropic-ai and contains 1.6MB of URLs, providing a comprehensive list of pages for crawling.
• Site built on Dealer.com platform with structured templates — The site uses the Dealer.com (DDC) platform, which typically provides structured data and templates that can be optimized for AI visibility.