australiansuper.com — AI Site Grade

Analysis

AustralianSuper: Akamai-Walled Fortress with Zero AI Crawler Access

Every AI crawler tested — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Applebot-Extended, and anthropic-ai — receives a 403 Access Denied from AkamaiGHost on every URL tested, including the homepage, /robots.txt, /sitemap.xml, /about-us, and /member. The site does not have a robots.txt file accessible to any crawler (it 403s), no llms.txt (also 403s), and no sitemap available. The only way to view the site's content is through Wayback Machine snapshots, which reveal a rich, well-structured site that AI models cannot reach.

Crawler Access

The entire domain sits behind Akamai (AkamaiGHost server, Akamai NS records) with a WAF that blocks all non-browser traffic uniformly. All 11 bot UAs plus a plain browser UA return identical 403 responses with Akamai reference IDs. The robots.txt file — the standard mechanism for communicating crawl permissions — is itself blocked at the WAF level, meaning no crawler can even read the fund's crawl preferences. The portal.australiansuper.com subdomain is similarly blocked. This is not a selective AI-bot block; it is a blanket denial of all programmatic access.

Cold-Knowledge Gap

The cold LLM knows AustralianSuper as "Australia's largest superannuation fund, managing over AUD $300 billion for more than 3 million members" — but the actual site (via Wayback) states $410 billion and over 3.6 million members. The cold knowledge is roughly 18 months stale on assets under management and membership numbers. The LLM also recalls "regulatory scrutiny from APRA in 2023" and "a class action regarding underpayment of death benefits" — negative signals that the site itself cannot counter-narrate to AI crawlers because they cannot reach it.

Schema Posture

The homepage (via Wayback) contains a single Organization JSON-LD schema with name, address, and telephone. No FinancialService, SuperannuationFund, FAQPage, Product, or WebSite schema types are present. The "why choose us" and "about us" pages have zero JSON-LD schema at all. Given the fund manages $410B and offers multiple investment options, insurance products, and pension products, the absence of product-level and FAQ schema is a significant missed signal for AI answer engines.

External Signals

DuckDuckGo search returned zero results for queries targeting australiansuper.com — the site has no organic search presence in that engine. The cold LLM knowledge contains reputational negatives (APRA scrutiny, class action) that the site's own content (which emphasizes awards, trust rankings, and member outcomes) cannot counterbalance in AI training data because crawlers cannot ingest it. The site's actual content is rich in comparison tables, fee calculators, and performance data — exactly the structured information AI answer engines prize — but none of it is accessible.

Findings

1. Akamai WAF blocks all AI crawlers uniformly High

   Every AI crawler tested (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Applebot-Extended, anthropic-ai) receives a 403 Access Denied from AkamaiGHost on all URLs, including the homepage, robots.txt, sitemap.xml, and subpages. The site has no accessible robots.txt or llms.txt.

   What to change: Allow AI crawler user agents (e.g., GPTBot, OAI-SearchBot, Google-Extended) through the Akamai WAF, and serve a robots.txt that permits crawling of public content.

2. Robots.txt is inaccessible to all crawlers High

   The robots.txt file returns a 403 error for all user agents, preventing crawlers from learning crawl permissions. This is a fundamental violation of the robots exclusion protocol.

   What to change: Serve a publicly accessible robots.txt that allows AI crawlers to access public pages.

3. Sitemap.xml is blocked by WAF High

   The sitemap.xml file returns a 403 error, preventing crawlers from discovering the site's URL structure.

   What to change: Serve a publicly accessible sitemap.xml listing all public pages.

4. No llms.txt file available Medium

   The llms.txt file returns a 403, meaning AI models have no guidance on which pages to use for training or retrieval.

   What to change: Create and serve an llms.txt file that lists key public pages for AI consumption.

5. Zero organic search presence on DuckDuckGo High

   Multiple search queries for australiansuper.com returned zero results on DuckDuckGo, indicating the site is not indexed by that search engine.

   What to change: Ensure the site is accessible to search engine crawlers and submit sitemaps to search engines.

6. Cold LLM knowledge is stale and contains negative signals High

   The cold LLM knows AustralianSuper as managing $300B with 3 million members, but the site states $410B and 3.6 million members. The LLM also recalls APRA scrutiny and a class action, which the site cannot counter because crawlers are blocked.

   What to change: Allow AI crawlers to access the site so LLMs can ingest current, positive content to counterbalance negative signals.

7. Missing FinancialService and SuperannuationFund schema Medium

   The homepage only has Organization schema; no FinancialService, SuperannuationFund, FAQPage, Product, or WebSite schema is present. The 'why choose us' and 'about us' pages have no JSON-LD schema at all.

   What to change: Add FinancialService, SuperannuationFund, FAQPage, and Product schema to relevant pages to improve AI answer engine understanding.

8. No FAQPage schema on help pages Medium

   The help-and-support page could not be fetched, but the site likely lacks FAQ schema, which is commonly used by AI answer engines.

   What to change: Add FAQPage schema to help and support pages.

9. Portal subdomain also blocked Medium

   The portal.australiansuper.com subdomain returns 403 for browser requests, indicating the entire domain ecosystem is blocked.

   What to change: Ensure public-facing subdomains are accessible to authorized crawlers.

What's working

• Rich, well-structured content visible in Wayback Machine — Wayback snapshots reveal a well-structured site with comparison tables, fee calculators, and performance data that would be valuable for AI answer engines if accessible.
• Organization schema present on homepage — The homepage includes Organization JSON-LD schema with name, address, and telephone, providing basic entity information.
• Strong brand recognition in cold LLM knowledge — The cold LLM correctly identifies AustralianSuper as Australia's largest superannuation fund with over $300B in assets and 3 million members, indicating strong brand presence in training data.