simplybusiness.co.uk — AI Site Grade

Analysis

The robots.txt and llms.txt both silently redirect to the homepage — meaning Simply Business has no AI-crawler governance whatsoever, yet every major AI bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot) gets a full 200 with real content.

Crawler Access

The site runs on WordPress VIP behind Cloudflare with AWS DNS. compare_bot_access on the homepage shows GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, ChatGPT-User, Perplexity-User, and Applebot-Extended all return 200 with ~228KB of content — identical to browser baseline. Bytespider is 403-blocked by Cloudflare. anthropic-ai (the training crawler) gets a 429 rate-limit block, meaning Anthropic's training pipeline is blocked while ClaudeBot (the real-time product crawler) passes. The robots.txt at simplybusiness.co.uk/robots.txt does not exist — it redirects to the homepage with zero rules. The llms.txt at simplybusiness.co.uk/llms.txt also redirects to the homepage. There is no AI-crawler directive file anywhere.

Cold-Knowledge Gap

The LLM knows Simply Business as a Travelers-owned digital broker (acquired 2017 for ~£190M) with a Trustpilot 4.6/5 rating and a policy management app. The actual site never mentions the Travelers acquisition on the homepage, business-insurance page, or landlord page — only the About Us page states it. The site cites a Feefo 4.5/5 rating (40k reviews), not Trustpilot. The LLM's prior about a "Simply Business app" for policy management does not appear as a prominent feature anywhere on the crawled pages. The site claims "nearly one million customers" and "over 900 employees" — the LLM prior had no employee count.

Schema Posture

The homepage carries rich JSON-LD including InsuranceAgency, Organization, PostalAddress, OpeningHoursSpecification, and SearchAction. The business-insurance and landlord-insurance pages add Product with AggregateRating (4.5/5, 40,103 reviews) and FAQPage with actual Q&A entities. The FAQ page itself is typed as both WebPage and FAQPage. Knowledge-centre articles use Article and NewsArticle schemas with author, datePublished, and wordCount. This is a strong schema implementation — the site is well-structured for AI extraction.

Content & Structure

The homepage is a 955-word JS-interactive page with a trade-search widget that returns "Loading... Sorry, that's not a trade we know!" when queried without JavaScript — a thin-content risk for crawlers that don't execute JS. The business-insurance page is a deep 3,996-word guide with real price examples (£5.76/month), comparison tables, and a claims statistic (£57 million paid out in 2025). The landlord-insurance page is similarly rich at 4,535 words. The knowledge centre (accessible via knowledge-sitemap.xml) contains hundreds of articles across categories like starting-out, business-tax, marketing, and retail — last updated May 2026.

External Signals

The sitemap.xml at the root redirects to the homepage (no content). The actual sitemap lives at wp-sitemap.xml (which redirects to sitemap_index.xml) and contains five sub-sitemaps including a 307KB knowledge-sitemap.xml and a 249KB page-sitemap.xml. The Feefo reviews page at feefo.com returns 403 to plain GET (JS-walled), so AI crawlers cannot independently verify the 4.5/40k rating claim. DNS TXT records confirm OpenAI domain verification (openai-domain-verification=dv-...) and Anthropic domain verification (anthropic-domain-verification-...), indicating the brand has proactively registered with both AI vendors — yet has no robots.txt or llms.txt to guide them.

Findings

1. No robots.txt file exists High

   The robots.txt URL redirects to the homepage with zero rules, meaning no AI crawler directives are in place.

   What to change: Create a robots.txt file at the root that explicitly allows or disallows AI crawlers as desired.

2. No llms.txt file exists High

   The llms.txt URL redirects to the homepage, providing no guidance to AI crawlers about which content to use.

   What to change: Create an llms.txt file listing key pages and any usage preferences for AI crawlers.

3. Anthropic training crawler is blocked Medium

   The anthropic-ai crawler receives a 429 rate-limit block, preventing Anthropic from using the site for model training while ClaudeBot (real-time) passes.

   What to change: Review rate-limiting rules to ensure desired AI crawlers are not inadvertently blocked.

4. Bytespider is blocked by Cloudflare Low

   Bytespider receives a 403 from Cloudflare, preventing any crawling by this AI bot.

   What to change: If Bytespider access is desired, adjust Cloudflare WAF rules to allow it.

5. Homepage is JS-dependent for core functionality Medium

   The homepage's trade-search widget returns a loading message when queried without JavaScript, creating thin content for crawlers that do not execute JS.

   What to change: Ensure the trade-search widget degrades gracefully with server-rendered fallback content.

6. Feefo reviews page is inaccessible to AI crawlers Medium

   The Feefo reviews page returns a 403 to plain GET requests, preventing AI crawlers from independently verifying the 4.5/40k rating claim.

   What to change: Work with Feefo to allow AI crawler access to the reviews page, or host a static copy on the site.

7. Root sitemap.xml redirects to homepage Low

   The standard sitemap.xml URL redirects to the homepage, though the actual sitemap exists at wp-sitemap.xml.

   What to change: Redirect sitemap.xml to the correct sitemap index or serve it directly.

8. Travelers acquisition not mentioned on key pages Low

   The homepage, business-insurance, and landlord-insurance pages do not mention the Travelers acquisition, though the About Us page does.

   What to change: Add a mention of the Travelers acquisition on the homepage and main product pages to align with LLM knowledge.

What's working

• Comprehensive JSON-LD schema across key pages — The site uses InsuranceAgency, Organization, Product with AggregateRating, FAQPage, and Article schemas, making content easily extractable by AI.
• Deep, informative content on product pages — Business-insurance and landlord-insurance pages contain thousands of words with real price examples, comparison tables, and claims statistics.
• Extensive knowledge centre with structured articles — Hundreds of articles across categories like starting-out and business-tax, using Article and NewsArticle schemas with author and date.
• OpenAI and Anthropic domain verification in place — DNS TXT records confirm the site has registered with OpenAI and Anthropic, indicating proactive engagement with AI vendors.
• FAQ page uses FAQPage schema with Q&A entities — The FAQ page is typed as both WebPage and FAQPage with actual questions and answers, aiding AI extraction.
• Major AI real-time crawlers have full access — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, and others receive 200 responses with full content.