What is Noindex? (Meta Noindex, Robots Noindex)

Learn what noindex is, how it prevents search engine indexing, and its growing role in controlling AI crawler access to your content.

A directive that tells search engines not to include a specific page in their index, preventing it from appearing in search results.

Noindex is an HTML meta tag or HTTP header that instructs search engine crawlers to exclude a page from their index. While the page can still be crawled and its links followed, it won't appear in search results. This directive has gained new significance as AI companies deploy crawlers to gather training data and power retrieval systems.

Deep Dive

The noindex directive gives publishers granular control over which pages appear in search results. Unlike robots.txt, which blocks crawling entirely, noindex allows search engines to access and analyze a page while keeping it out of public results. This distinction matters for link equity: a noindexed page can still pass authority to other pages it links to. Implementation typically happens in one of two ways. The meta robots tag goes in the HTML head: `<meta name="robots" content="noindex">`. Alternatively, the X-Robots-Tag HTTP header achieves the same result for non-HTML files like PDFs. You can also target specific crawlers using their names: `<meta name="googlebot" content="noindex">` only affects Google while leaving Bing unaffected. Common use cases include thank-you pages after form submissions, paginated archives, staging environments, and duplicate content that serves users but shouldn't compete in search. E-commerce sites often noindex filter result pages that create thousands of low-value URLs. News sites might noindex syndicated content to avoid canonical confusion. The AI angle adds complexity. AI crawlers like GPTBot and ClaudeBot don't necessarily respect noindex the way traditional search engines do - they have their own directives and often rely on robots.txt instead. A page marked noindex for Google might still be scraped for AI training unless you explicitly block the AI crawlers separately. This creates a new consideration: do you want your content in traditional search, AI training data, both, or neither? For marketers, noindex remains essential for index hygiene. Google has a crawl budget - even for large sites - and wasting it on pages that shouldn't rank dilutes your indexing efficiency. More strategically, noindex helps prevent keyword cannibalization by keeping similar pages from competing against each other. The key is being deliberate: every page should either be worthy of ranking or explicitly excluded.

Why It Matters

Noindex is a precision tool for controlling your search presence. Without it, you're at the mercy of search engines deciding which of your pages matter - and they often get it wrong, indexing login pages, internal search results, and parameter variations that cannibalize your real content. The stakes are higher now that AI systems are actively scraping the web. Your noindex strategy needs to account for both traditional search visibility and AI training data access. A page you never wanted ranking might end up informing how ChatGPT describes your brand. Understanding noindex - and its limitations - is essential for managing your content's presence across both search engines and AI systems.

Examples

During a technical SEO audit: We found 15,000 noindexed pages that are still in the sitemap. That's confusing Google - we're basically saying 'here are important pages' and 'don't index these' simultaneously.

In a content strategy discussion: Let's noindex the old event pages instead of deleting them. Attendees might still need that information, but we don't want them competing with this year's event page.

When discussing AI training data: Just because we noindexed our gated content doesn't mean AI crawlers aren't scraping it. We need to block GPTBot and ClaudeBot in robots.txt if we want to keep it out of their training sets.

Common Misconceptions

Misconception: Noindex prevents Google from seeing the page entirely. Reality: Google still crawls noindexed pages and follows their links. The page is processed and analyzed - it just won't appear in search results. This is why noindexed pages can still pass PageRank to pages they link to.

Misconception: Noindex and robots.txt disallow do the same thing. Reality: They're fundamentally different. Robots.txt blocks crawling entirely - Google never sees the content. Noindex allows crawling but prevents indexing. For maximum exclusion, you actually need robots.txt, not noindex.

Misconception: Adding noindex will quickly remove a page from search results. Reality: Google must recrawl the page to discover the noindex directive. For pages that are rarely crawled, this can take weeks or months. Request removal via Search Console for faster results.

Key Takeaways

Noindex hides pages from search results, not crawlers: Pages can still be crawled, analyzed, and pass link equity. They simply won't appear in search results. This is different from blocking via robots.txt.

AI crawlers often ignore noindex directives: GPTBot, ClaudeBot, and similar AI crawlers may not respect meta noindex tags. Blocking them requires separate robots.txt rules targeting each crawler specifically.

Overuse wastes crawl budget and dilutes authority: Every noindexed page still consumes crawl resources. Better to prevent creation of low-value pages than to noindex thousands of them after the fact.

Implementation method matters for non-HTML content: PDFs, images, and other files can't contain meta tags. Use X-Robots-Tag HTTP headers to noindex these file types instead.

Related Terms

Robots.txt: Robots.txt blocks crawling at the directory level, while noindex allows crawling but prevents indexing of specific pages.

Indexing: Noindex is the primary mechanism for excluding pages from the indexing process while keeping them accessible to users.

AI Crawlers: AI crawlers like GPTBot often require separate blocking via robots.txt since they may not respect noindex meta tags.

Index Control Affects AI Visibility

Pages you've noindexed from traditional search can still influence AI responses if AI crawlers accessed them before blocking. Trakkr helps you understand where your brand actually appears in AI-generated responses, regardless of your indexing preferences - revealing gaps between your intended visibility strategy and reality.