citylit.ac.uk — AI Site Grade

Analysis

City Lit — AI-Visibility Audit

ClaudeBot receives a 403 Forbidden on the homepage while every other major AI crawler (GPTBot, PerplexityBot, Google-Extended, OAI-SearchBot, anthropic-ai) gets a full 200 with identical content — a selective Anthropic block that creates a blind spot for Claude-powered retrieval-augmented generation systems.

Crawler Access

The robots.txt at www.citylit.ac.uk/robots.txt contains no rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or OAI-SearchBot. The wildcard User-agent: * rule disallows /catalog/, /checkout/, /customer/, and filtered URL patterns, but course pages at /courses/ are fully crawlable. However, the server (Varnish cache behind Fastly CDN, IPs on 151.101.x.x) enforces a UA-level block on ClaudeBot at the edge — compare_bot_access shows ClaudeBot returns a 403 with a Varnish error page while anthropic-ai (the training crawler) passes through at 200. GPTBot is also blocked at 403 despite no robots.txt directive. This means Claude-powered RAG tools and GPT-powered search cannot retrieve the site's content live. The llms.txt returns a 404 — no AI-friendly content map exists.

Cold-Knowledge Gap

The LLM prior knows City Lit as a "specialist adult education college" offering "over 5,000 part-time courses" with a "Good" Ofsted rating from 2023. The site itself claims "Outstanding" Ofsted (May 2023) — a material discrepancy. The prior also understates the course catalogue: the site lists 2,526 courses across 9 departments, not 5,000. The prior mentions "inclusive access" and "learners with learning difficulties" but misses the Centre for Deaf Education, the Malorie Blackman Scholarships, the Mental Wealth Festival, and the City Lit Fellows (Grayson Perry, Gillian Anderson, Dame Mary Beard, Lulu, HRH The Princess Royal) — all prominently featured on the site.

Schema Posture

Individual course pages carry rich, well-structured Course schema with aggregateRating, Offer, CourseInstance, Schedule, and instructor Person entities. The homepage has EducationalOrganization schema with address, founding date (1919), and social profiles. The blog uses BlogPosting with datePublished/dateModified. However, the events page has no Event schema — it lists upcoming events (Pride at City Lit, Refugee Week, The Log Books talk) in plain HTML with no structured data. The Product schema on course pages uses a hardcoded priceValidUntil of "2024-07-31" — a stale date that may confuse price freshness signals.

External Signals

The site runs on Adobe Commerce (Magento) with a WilliamsCommerce custom theme, hosted on Fastly CDN with Varnish caching. DNS TXT records show integrations with Zoom, Stripe, Zendesk, Atlassian, Adobe, Apple, and Google — a mature tech stack. External search results for the brand are surprisingly sparse: DuckDuckGo returned zero results for "City Lit adult education London reviews" and "City Lit Ofsted outstanding 2023," suggesting limited off-domain citation density that could weaken AI-generated brand summaries. The site's blog is active (97 posts in Art & Design alone, posts dated through May 2026) and contains rich narrative content that AI crawlers could use for brand grounding — if they can access it.

Findings

1. ClaudeBot receives 403 Forbidden on homepage High

   ClaudeBot is blocked at the edge with a 403 error while other AI crawlers (including anthropic-ai) get full 200 responses. This prevents Claude-powered RAG tools from retrieving live content.

   What to change: Remove the ClaudeBot-specific block at the edge (Varnish/Fastly) to allow ClaudeBot access, matching the treatment of anthropic-ai.

2. GPTBot receives 403 Forbidden on homepage High

   GPTBot is also blocked with a 403 despite no robots.txt disallowance, preventing GPT-powered search from retrieving live content.

   What to change: Remove the GPTBot-specific block at the edge to allow GPTBot access.

3. llms.txt returns 404 Medium

   The site does not provide an llms.txt file, missing an opportunity to offer an AI-friendly content map.

   What to change: Create an llms.txt file at the root listing key pages (courses, about, blog) to guide AI crawlers.

4. Events page lacks Event schema Medium

   The events page lists upcoming events in plain HTML with no structured data, reducing visibility in AI-generated event summaries.

   What to change: Add Event schema markup to each event listing on the events page.

5. Course schema uses stale priceValidUntil date Medium

   The Product schema on course pages hardcodes priceValidUntil as '2024-07-31', a past date that may confuse price freshness signals for AI crawlers.

   What to change: Update priceValidUntil to a future date or remove it if prices do not expire.

6. LLM prior states 'Good' Ofsted, site claims 'Outstanding' Medium

   The LLM prior recalls a 'Good' Ofsted rating from 2023, but the site prominently claims 'Outstanding' (May 2023). This discrepancy may cause AI summaries to understate the institution's quality.

   What to change: Ensure the Ofsted rating is prominently marked up with schema (e.g., EducationalOrganization.aggregateRating) and referenced in authoritative external sources.

7. Low external citation density for brand Medium

   Web searches for City Lit reviews and Ofsted ratings returned zero results, indicating limited off-domain mentions that could weaken AI-generated brand summaries.

   What to change: Encourage student reviews on third-party platforms and ensure Ofsted rating is listed on official directories.

8. LLM prior overstates course count Low

   The LLM prior claims 'over 5,000 part-time courses', but the site lists 2,526 courses across 9 departments. This inflation may mislead AI-generated descriptions.

   What to change: Ensure the site's course count is accurately reflected in structured data and visible to crawlers.

9. LLM prior misses key brand elements Low

   The prior lacks mention of the Centre for Deaf Education, Malorie Blackman Scholarships, Mental Wealth Festival, and City Lit Fellows, all prominently featured on the site.

   What to change: Add structured data (e.g., EducationalOrganization.owns) to explicitly associate these elements with the institution.

What's working

• Course pages have well-structured Course schema — Individual course pages include Course schema with aggregateRating, Offer, CourseInstance, Schedule, and instructor Person entities, providing rich data for AI crawlers.
• Homepage has EducationalOrganization schema — The homepage includes EducationalOrganization schema with address, founding date (1919), and social profiles, establishing institutional identity for AI.
• Blog uses BlogPosting schema with dates — Blog posts include BlogPosting schema with datePublished and dateModified, aiding AI in understanding content freshness.
• Active blog with rich narrative content — The blog has 97 posts in Art & Design alone, with posts dated through May 2026, providing rich content for AI grounding.
• Course pages are crawlable by most AI bots — Despite selective blocks, most AI crawlers (including anthropic-ai, PerplexityBot, Google-Extended, OAI-SearchBot) can access course pages with full content.
• Sitemap includes course product URLs — The sitemap index contains a product sitemap with course URLs, aiding discovery by crawlers.
• Mature tech stack with CDN and integrations — The site uses Fastly CDN, Varnish caching, and integrates with Zoom, Stripe, Zendesk, etc., indicating a well-maintained infrastructure.