healthsherpa.com — AI Site Grade

Analysis

Every AI Crawler Gets a 406

HealthSherpa.com returns HTTP 406 Not Acceptable to every single user-agent tested — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, ChatGPT-User, Applebot-Extended, and even a standard browser — from every URL path attempted, including the homepage, /robots.txt, /llms.txt, /find_plans, /about, /blog, and /privacy_notice. The site is entirely inaccessible to AI crawlers and search engines alike.

Crawler Access

The site sits behind a custom spaces-router (header via: 1.1 spaces-router (45109b4b6ff5)) that rejects all plain HTTP requests with a 20-byte "406 Not Acceptable" response. No robots.txt exists — or if it does, it cannot be retrieved. No llms.txt exists. DNS points to AWS (100.24.247.153, 32.195.75.96, 50.16.162.94) with no CDN layer like Cloudflare. The anthropic-domain-verification TXT record is present, confirming the brand has engaged with Anthropic for crawler verification, but the actual site still blocks ClaudeBot at the router level.

Cold-Knowledge Gap

LLMs know HealthSherpa as a top third-party ACA enrollment platform founded in 2013, processing millions of enrollments, with a broker tool called "HealthSherpa for Agents" and multilingual support. The Wayback Machine snapshot from January 2025 confirms this: the live site was a fully functional enrollment portal with plan comparison, subsidy calculators, and agent tools. The current live site, however, delivers zero content to any non-browser client. The gap is absolute — AI models describe a working marketplace that the actual domain no longer serves to automated agents.

Schema Posture

The January 2025 snapshot contained valid Organization and WebSite JSON-LD schema with legal name "HealthSherpa.com", a customer service phone number +1 (855) 772-2663, social profiles (Facebook, Twitter, LinkedIn), and a SearchAction targeting zip-code-based plan lookup. The current live site serves no schema at all — it serves no content at all. The schema that exists in the Wayback Machine is orphaned from the live domain.

External Signals

DuckDuckGo returns zero indexed results for site:healthsherpa.com, "HealthSherpa", or any combination of the brand name with "ACA" or "insurance". The site has been effectively de-indexed from at least one major search engine. The Wayback Machine shows the site was operational and content-rich as recently as January 2025, suggesting the 406 blockade is a recent or ongoing configuration issue rather than a permanent architectural choice.

Findings

1. All AI crawlers and search engines receive HTTP 406 High

   Every tested user-agent (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, ChatGPT-User, Applebot-Extended, and standard browser) receives a 406 Not Acceptable response from every URL path, including the homepage, robots.txt, and all subpages. The site is entirely inaccessible to automated agents.

   What to change: Remove the blanket 406 rejection at the router level for legitimate AI crawlers and search engine bots. Allow access to static content and key pages while maintaining security for dynamic enrollment flows.

2. No accessible robots.txt file High

   The robots.txt file returns a 406 error, making it impossible for crawlers to discover allowed paths. No crawl directives are available.

   What to change: Serve a valid robots.txt file that allows AI crawlers and search engines to access public content.

3. No llms.txt file Medium

   The llms.txt file returns a 406 error, so AI models have no structured guidance about the site's content or resources.

   What to change: Create and serve an llms.txt file that lists key pages and resources for AI crawlers.

4. Zero search engine indexing High

   DuckDuckGo returns zero results for site:healthsherpa.com or any brand-related queries. The site is effectively de-indexed from at least one major search engine.

   What to change: Allow search engine crawlers to access the site and submit an updated sitemap to search engines.

5. No structured data on live site High

   The current live site serves no JSON-LD or other structured data because it serves no content at all. The schema from the January 2025 Wayback snapshot is orphaned.

   What to change: Restore Organization and WebSite JSON-LD schema on the live homepage, including legal name, contact info, and search action.

6. AI models describe a working site that no longer exists Medium

   LLMs describe HealthSherpa as a functional ACA enrollment platform with plan comparison and agent tools, but the live site delivers zero content to automated agents. This mismatch can cause AI-generated responses to be inaccurate or outdated.

   What to change: Allow AI crawlers to access the site so that AI models can retrieve current content and update their knowledge.

7. Custom router blocks all plain HTTP requests High

   The site uses a custom 'spaces-router' that rejects all plain HTTP requests with a 20-byte '406 Not Acceptable' response. No CDN layer like Cloudflare is present.

   What to change: Reconfigure the router to allow access for legitimate crawlers and search engines, or add a CDN layer that can differentiate between bots and malicious traffic.

8. Anthropic domain verification present but ClaudeBot still blocked Medium

   The DNS TXT record includes anthropic-domain-verification, indicating engagement with Anthropic for crawler verification, yet ClaudeBot still receives a 406 error.

   What to change: Ensure that verified crawlers like ClaudeBot are allowed through the router after domain verification.

What's working

• Wayback snapshot confirms a content-rich site existed in January 2025 — The January 2025 snapshot shows a fully functional enrollment portal with plan comparison, subsidy calculators, and agent tools, indicating the site has valuable content that can be restored for AI visibility.
• Historical JSON-LD schema is valid and comprehensive — The January 2025 snapshot contained valid Organization and WebSite JSON-LD schema with legal name, phone number, social profiles, and a SearchAction, providing a template for restoring schema on the live site.
• Anthropic domain verification TXT record is present — The DNS includes an anthropic-domain-verification record, showing proactive engagement with AI crawler verification, which can be leveraged once access is granted.
• LLMs recognize HealthSherpa as a top ACA enrollment platform — AI models have knowledge of HealthSherpa as a leading third-party ACA marketplace with millions of enrollments, broker tools, and multilingual support, providing a strong foundation for AI visibility once the site is accessible.