Technical & Foundational Health
Technical checks look at whether search engines and AI crawlers can reach, read and index your pages: robots.txt, sitemaps, status codes, canonical tags, indexing directives, HTTPS, server headers and page speed. If crawlers cannot open a page, or are told not to index it, nothing else on the page can help it appear in search results or AI answers.
Crawlability Checks
Whether crawlers can reach your site, read your rules and your sitemap, and follow your links
robots.txt Validation and AI Bot Permissions
User-agent: * rules, so you do not need an Allow line for every bot.
Allow: /, or remove the rule that blocks it, and repeat your private-path Disallow lines in that group (a named group does not inherit the * rules). Decide training access separately. The AI-Specific check reference covers AI Bot Permissions in full.
# robots.txt - allow AI search, decline training (example policy) User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / Disallow: /admin/ User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended Disallow: / User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Sitemap Validation
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2024-01-15</lastmod>
</url>
<url>
<loc>https://example.com/about</loc>
<lastmod>2024-01-10</lastmod>
</url>
</urlset>
Sitemap Membership
Site Crawlability
JS Rendering
Redirect Chain
Broken Links
Indexation Checks
Whether search engines may index the page, show its text and keep the address you want
Google Index Status
Bing Index Status
Canonical URL
<head> <link rel="canonical" href="https://example.com/page-url"> </head>
Canonical Target
Soft 404
X-Robots-Tag
Snippet Controls
Security Checks
Secure connections and resources, for visitors and crawlers
HTTPS Security
Mixed Content
Link Security
Content Security Policy
Performance Checks
Speed and server settings. Compression, HTTP/2 Support and Cache Headers read the response headers of a request for the page. The separate Performance dimension scores the PageSpeed lab metrics in detail: see Performance checks.
Page Load Speed
Core Web Vitals
Compression
HTTP/2 Support
Resource Hints
<head> <!-- Preconnect to critical origins --> <link rel="preconnect" href="https://fonts.googleapis.com"> <link rel="preconnect" href="https://cdn.example.com" crossorigin> <!-- DNS prefetch for other domains --> <link rel="dns-prefetch" href="https://analytics.example.com"> <!-- Preload critical resources --> <link rel="preload" href="/fonts/main.woff2" as="font" type="font/woff2" crossorigin> </head>
Cache Headers
Compatibility Checks
Phones, languages and translated versions
Mobile Responsiveness
<!-- Add to <head> -->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- CSS media queries -->
<style>
@media (max-width: 768px) {
.container { padding: 10px; }
.sidebar { display: none; }
}
</style>
Hreflang Tags
<head> <link rel="alternate" hreflang="en" href="https://example.com/page"> <link rel="alternate" hreflang="es" href="https://example.com/es/page"> <link rel="alternate" hreflang="fr" href="https://example.com/fr/page"> <link rel="alternate" hreflang="x-default" href="https://example.com/page"> </head>
Language Attribute
Link Previews and llms.txt
Tags that build the preview of a shared link, and the optional llms.txt file. The former AI Meta Tags check no longer runs: no search or AI engine documents those tags.
llms.txt Implementation
# Example Company > Leading provider of innovative solutions in the tech industry. ## Important pages - [About](https://example.com/about): Company overview and mission - [Products](https://example.com/products): Product catalog and specifications - [Blog](https://example.com/blog): Technical articles and industry insights ## Contact - Email: contact@example.com
OpenGraph Tags
<head> <meta property="og:title" content="Your Page Title"> <meta property="og:description" content="A short description of what the page offers"> <meta property="og:image" content="https://example.com/image.jpg"> <meta property="og:image:width" content="1200"> <meta property="og:image:height" content="630"> <meta property="og:url" content="https://example.com/page"> <meta property="og:type" content="article"> <meta property="og:site_name" content="Example"> </head>
Twitter Cards
<head> <meta name="twitter:card" content="summary_large_image"> <meta name="twitter:title" content="Your Page Title"> <meta name="twitter:description" content="A compelling description"> <meta name="twitter:image" content="https://example.com/image.jpg"> </head>