mcgovernauto.com — AI Site Grade

Analysis

The anthropic-ai bot is the only AI crawler that can see this site

McGovern Automotive Group operates a massive multi-franchise dealership network across MA, NH, NJ, and NY, yet its website is invisible to every major AI crawler except Anthropic's. The robots.txt explicitly disallows GPTBot, OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, CCBot, Baiduspider, and Yandex from crawling content paths. But the deeper problem is a two-tier WAF blockade: AkamaiGHost returns HTTP 403 to every browser and every bot user-agent tested — except anthropic-ai, which passes through to an nginx backend serving full 1MB+ pages. Google-Extended, Applebot-Extended, and Bytespider all get blocked at the edge before even reaching robots.txt.

Crawler Access

The site uses Akamai WAF (edgesuite.net) as a reverse proxy. All 11 bot user-agents tested — including GPTBot, Google-Extended, ClaudeBot, PerplexityBot, OAI-SearchBot, ChatGPT-User, Bytespider, and Applebot-Extended — receive HTTP 403 "Access Denied" with a 364-byte HTML shell. Only anthropic-ai gets HTTP 200 and full page content (1,056,551 bytes). The robots.txt is also blocked by the WAF for non-Anthropic bots, meaning crawlers that hit the 403 wall never even see the disallow rules. The llms.txt returns 403 for all user-agents. No llms.txt exists.

Content & Schema Posture

The homepage is a heavy JS-rendered single-page application built on the Dealer.com (CDK Global) platform. Visible text content is loaded dynamically — a plain GET returns only 15 words. The Wayback Machine snapshot reveals 4,863 words of actual content including a massive inventory listing spanning 30+ brands (Porsche, Ferrari, Maserati, BMW, Audi, Toyota, Ford, Chevrolet, Hyundai, Subaru, etc.) across 50+ dealership locations. JSON-LD schema is present but fragmented: the homepage carries a Product schema with an AggregateRating (4.49/5 from 22,218 reviews) and an Organization schema listing only one sub-organization (McGovern Chevrolet of Plymouth) — despite the site representing 50+ dealerships. The about-us.htm page returns HTTP 404. The sitemap contains thousands of URLs with lastmod dates as recent as 2026-05-30.

Cold-Knowledge Gap

The LLM's prior knowledge describes McGovern Auto as a "family-owned Massachusetts dealership group" with "Chevrolet, Buick, GMC, Ford, and Hyundai" franchises founded by "John McGovern in the 1970s." The actual site reveals a far larger operation: 50+ dealerships spanning luxury brands (Porsche, Ferrari, Maserati, Alfa Romeo, Land Rover, Jaguar, Genesis, Audi, BMW, Mercedes-Benz) across four states (MA, NH, NJ, NY). The cold knowledge misses the entire luxury portfolio, the Newton headquarters address (777 Washington St, Newton, MA 02460), the scale of the used inventory operation, and the "McGovern Certified" program. The prior knowledge is 3-5 years out of date relative to the current site footprint.

External Signals

Web search returns zero indexed results for "McGovern Auto Group" across multiple query variations — a striking absence suggesting the site's Akamai WAF blockade has suppressed organic search visibility. The site carries 22,218 aggregated reviews (4.49/5) in its own schema, but no external review platforms surfaced in search results. DNS records point to dealer.com nameservers and cdkglobal.com SPF records, confirming the CDK Global / Dealer.com platform. The site's facebook-domain-verification and google-site-verification meta tags are present, indicating past search console setup that may have degraded under the WAF blockade.

Findings

1. Akamai WAF blocks all AI crawlers except anthropic-ai High

   The site's Akamai WAF returns HTTP 403 to every tested bot user-agent (GPTBot, Google-Extended, ClaudeBot, PerplexityBot, OAI-SearchBot, ChatGPT-User, Bytespider, Applebot-Extended) except anthropic-ai, which receives full page content.

   What to change: Reconfigure the Akamai WAF to allow major AI crawlers (GPTBot, Google-Extended, ClaudeBot, PerplexityBot, OAI-SearchBot) to access the site, or serve static HTML versions to bots.

2. robots.txt blocked by WAF for non-Anthropic bots High

   The robots.txt file returns HTTP 403 for all user-agents except anthropic-ai, preventing crawlers from seeing disallow rules and effectively blocking access.

   What to change: Ensure robots.txt is publicly accessible without WAF authentication for all user-agents.

3. llms.txt file missing Medium

   The llms.txt file returns HTTP 403 for all user-agents, indicating it does not exist or is blocked.

   What to change: Create an llms.txt file with a summary of the site's content and links to key pages for AI crawlers.

4. Homepage is a heavy JS-rendered SPA High

   The homepage returns only 15 words of text in a plain GET request; actual content (4,863 words) is loaded dynamically via JavaScript, making it invisible to crawlers that do not execute JS.

   What to change: Implement server-side rendering or pre-rendered static HTML for key pages to ensure content is accessible to all crawlers.

5. About Us page returns HTTP 404 Medium

   The about-us.htm page returns a 404 error, which is a key page for brand information and AI knowledge.

   What to change: Restore the About Us page with accurate and comprehensive information about the dealership group.

6. JSON-LD schema is fragmented and incomplete Medium

   The homepage contains a Product schema with an AggregateRating but the Organization schema lists only one sub-organization (McGovern Chevrolet of Plymouth) despite the site representing 50+ dealerships.

   What to change: Expand the Organization schema to include all dealership locations and brands, and ensure LocalBusiness schemas are present for each location.

7. Zero indexed pages in web search results High

   Multiple web searches for 'McGovern Auto Group' and related terms returned zero results, indicating the site is not indexed by search engines, likely due to the WAF blockade.

   What to change: Allow search engine bots (Googlebot, Bingbot) through the WAF and ensure content is crawlable and indexable.

8. LLM prior knowledge is 3-5 years out of date Medium

   The LLM's knowledge describes McGovern Auto as a small Massachusetts group with only a few brands, missing the current 50+ dealerships, luxury brands, and multi-state footprint.

   What to change: Publish up-to-date information on the website and ensure it is accessible to AI crawlers to refresh knowledge.

9. No external review platform presence found Low

   Web searches for reviews on DealerRater and other platforms returned zero results, despite the site claiming 22,218 reviews.

   What to change: Encourage customers to leave reviews on major platforms and ensure those pages are indexed.

What's working

• Anthropic AI crawler has full access — The anthropic-ai bot receives HTTP 200 and full page content (1MB+), allowing Claude to index the site's content.
• Sitemap accessible to anthropic-ai with thousands of URLs — The sitemap.xml is served to anthropic-ai and contains thousands of URLs with recent lastmod dates, providing a comprehensive inventory of pages.
• JSON-LD Product schema with AggregateRating present — The homepage includes a Product schema with an AggregateRating of 4.49/5 from 22,218 reviews, which can enhance rich snippets.
• Google and Facebook verification meta tags present — The site includes google-site-verification and facebook-domain-verification meta tags, indicating past search console and social media setup.
• Massive inventory of 30+ brands across 50+ locations — The site lists a vast inventory spanning luxury and mainstream brands across four states, providing rich content for AI to index.