crafterscompanion.co.uk — AI Site Grade

Analysis

Cloudflare Wall Blocks Every AI Crawler From Real Content

Every major AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, ChatGPT-User, OAI-SearchBot, Applebot-Extended — receives a 403 Cloudflare JS-challenge page when requesting the homepage. The robots.txt contains zero AI-specific rules (no mention of any AI bot), so the blocking is entirely at the CDN/WAF layer, not via directive. The sitemap at /sitemap.xml also returns 403 to automated requests. The site is hosted on Shopify (23.227.38.65) behind Cloudflare's managed challenge mode, which requires JavaScript execution and cookie acceptance before serving any page content.

Cold-Knowledge Gap: Insolvency History the Site Conceals

LLM cold knowledge recalls that Crafter's Companion entered administration in September 2023 and was acquired by a new entity, "Crafter's Companion Ltd," in October 2023. The current live site contains zero mention of this event. The archived January 2024 About Us page (Wayback Machine) describes a company with "more than 200 staff" and a founding story by Sara Davies, but the live site's homepage text excerpt — the only content AI crawlers can reach through the Cloudflare wall — is a thin 183-word product listing with no company narrative, no history, and no acknowledgment of the ownership change. The Organization schema on the homepage lists the company as "Crafters Companion UK" with a registered address in Co. Durham, but the company number and registration status are absent from the schema.

llms.txt Exists but Is Inaccessible to Bots

The site has a remarkably sophisticated llms.txt file that implements the Universal Commerce Protocol (UCP) and directs AI agents to use the Shop skill (shop.app/SKILL.md) for transactions. It provides structured endpoints for product discovery, cart creation, and checkout. However, the agents.md file it references, the UCP discovery endpoint at /.well-known/ucp, and every product/collection page all return the same Cloudflare 403 challenge page. The llms.txt itself was accessible (200), but the actual agent-facing infrastructure it describes is unreachable by any automated agent that cannot execute JavaScript.

Schema and Content Signals

The homepage carries Organization and WebSite schema with correct name, URL, address, contact phone (01388663251), and social links (Facebook, Instagram, YouTube). A SearchAction potentialAction is present. The page has FAQ signals and list structures but no FAQ schema markup — the FAQ content is plain HTML. Heading structure is flat (all H2/H3, no H1 on the homepage excerpt). The site has a blog (/blogs/inspiration/) and a "Crafter's TV" section, but both are behind the Cloudflare wall. The archived sitemap shows roughly 2,000+ product URLs across two product sitemaps, plus collections, pages, and blogs — a substantial catalog that AI crawlers cannot inventory.

Findings

1. Cloudflare JS-challenge wall blocks every major AI crawler High

   Every major AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, ChatGPT-User, OAI-SearchBot, Applebot-Extended) receives a 403 Cloudflare JS-challenge page when requesting the homepage. The robots.txt contains no AI-specific rules, so blocking is entirely at the CDN/WAF layer.

   What to change: Configure Cloudflare to allow AI crawler user agents (e.g., GPTBot, ClaudeBot) through the WAF, or serve a static HTML version of the site to these bots.

2. Sitemap returns 403 to automated requests High

   The sitemap at /sitemap.xml returns a 403 Cloudflare challenge page, preventing AI crawlers from discovering the site's URL inventory.

   What to change: Allow access to /sitemap.xml for all user agents, or serve it from a separate domain not behind Cloudflare challenge.

3. llms.txt infrastructure unreachable by AI agents High

   The site has a sophisticated llms.txt implementing Universal Commerce Protocol, but the referenced agents.md, UCP discovery endpoint (/.well-known/ucp), and all product/collection pages return Cloudflare 403 challenges. The llms.txt itself is accessible, but the agent-facing infrastructure it describes is completely unreachable.

   What to change: Remove Cloudflare JS-challenge for AI crawler user agents, or serve the llms.txt infrastructure from a separate domain without challenge.

4. 2023 insolvency and ownership change concealed from AI knowledge Medium

   LLM cold knowledge recalls that Crafter's Companion entered administration in September 2023 and was acquired by a new entity. The live site contains zero mention of this event, and the homepage text excerpt (the only content AI crawlers can reach) is a thin 183-word product listing with no company narrative or ownership history.

   What to change: Add a clear statement about the company's history and current ownership on the About Us page and in Organization schema.

5. FAQ content lacks FAQ schema markup Medium

   The homepage contains FAQ content rendered as plain HTML with no FAQ schema markup, reducing the chance of appearing in AI-generated answers.

   What to change: Add FAQPage schema markup to the FAQ section on the homepage.

6. Homepage lacks H1 heading Low

   The homepage heading structure is flat with only H2 and H3 elements; no H1 heading is present, which weakens semantic structure for crawlers.

   What to change: Add a descriptive H1 heading to the homepage that includes the brand name and primary offering.

7. Organization schema missing company registration details Low

   The Organization schema on the homepage lists the company name and address but omits the company number and registration status, reducing trust signals for AI agents.

   What to change: Add the company registration number and registration authority to the Organization schema.

8. Entire product catalog behind Cloudflare wall High

   All product and collection pages return 403 Cloudflare challenges, preventing AI crawlers from indexing the 2,000+ product catalog.

   What to change: Allow AI crawler access to product and collection pages by configuring Cloudflare to bypass JS-challenge for known AI user agents.

9. Blog and Crafter's TV sections inaccessible to AI crawlers Medium

   The blog (/blogs/inspiration/) and Crafter's TV section are behind the Cloudflare wall, preventing AI agents from accessing rich content that could improve brand visibility.

   What to change: Allow AI crawler access to blog and TV pages by configuring Cloudflare to bypass JS-challenge for known AI user agents.

10. Robots.txt contains no AI-specific rules Low

    The robots.txt file has 5 user-agent rules but none mention any AI bot, missing an opportunity to explicitly guide AI crawlers.

    What to change: Add explicit rules for AI crawlers (e.g., GPTBot, ClaudeBot) in robots.txt to allow access to key pages.

What's working

• Sophisticated llms.txt implementing Universal Commerce Protocol — The site has a well-structured llms.txt that directs AI agents to use the Shop skill for transactions and provides structured endpoints for product discovery, cart creation, and checkout.
• Organization and WebSite schema present on homepage — The homepage carries Organization and WebSite schema with correct name, URL, address, phone, and social links, providing basic structured data for AI agents.
• SearchAction potentialAction in WebSite schema — The WebSite schema includes a SearchAction potentialAction, enabling AI agents to perform site searches directly.
• Archived sitemap reveals substantial product catalog — The archived sitemap from January 2024 shows over 2,000 product URLs across two product sitemaps, plus collections, pages, and blogs, indicating a rich catalog that could be indexed if access were allowed.