Technical & Foundational Health

15% Weight 23 Checks 5 Critical

Technical checks evaluate your website's infrastructure, security, crawlability, and performance. These foundational elements determine whether AI systems can access and properly index your content.

Critical checks first: Focus on fixing critical issues (robots.txt, HTTPS, crawlability) before addressing other technical optimizations.

Crawlability Checks

Ensure AI crawlers can access your content

Robots.txt AI Bot Access

Critical ▼

What It Checks

Analyzes your robots.txt file for permissions granted to AI crawlers including GPTBot, ChatGPT-User, Claude-Web, ClaudeBot, PerplexityBot, Bingbot, and other AI user agents.

Why It's Important

AI systems respect robots.txt directives. Blocking AI bots prevents your content from being indexed and cited in AI-generated responses. This is the #1 reason content doesn't appear in AI answers.

Scoring Method

100 points if all AI bots are allowed. Deducts 15 points per blocked bot. Score decreases significantly with multiple blocked bots.

Pass: ≤2 blocked bots Warning: 3-5 blocked Fail: >5 blocked

How To Fix

Add explicit Allow directives for AI bots in your robots.txt file. Place the file at the root of your domain (e.g., example.com/robots.txt).

Implementation Example

# Allow AI crawlers
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Bingbot
Allow: /

# Standard crawlers
User-agent: *
Allow: /

Sitemap Validation

High ▼

What It Checks

Verifies the existence and validity of your sitemap.xml file, checks for proper XML structure, URL count, and presence of priority and changefreq tags.

Why It's Important

Sitemaps help AI crawlers discover and prioritize your content. A well-structured sitemap ensures comprehensive content indexing and helps crawlers find important pages faster.

Scoring Method

100 points for a valid sitemap with all metadata (priority, changefreq). 90 for partial metadata. 0 if missing or blocked.

Pass: Valid sitemap exists Fail: No sitemap or invalid

How To Fix

Create an XML sitemap at /sitemap.xml with proper structure. Include all important URLs with lastmod, priority, and changefreq tags.

Implementation Example

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2024-01-15</lastmod>
    <changefreq>weekly</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2024-01-10</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

Site Crawlability

Critical ▼

What It Checks

Tests basic accessibility of your site including HTTP response codes, server availability, and detection of bot protection systems (Cloudflare, Akamai, etc.).

Why It's Important

If AI crawlers can't access your site, your content won't be indexed or cited. Bot protection that blocks legitimate crawlers prevents AI visibility entirely.

Scoring Method

100 points for 200 OK response. 30 points for bot protection detected (403). 0 for server errors (5xx).

Pass: 2xx responses Fail: 4xx/5xx errors

How To Fix

Ensure server uptime, whitelist known AI bot user agents in your WAF/CDN, fix broken pages, and handle errors gracefully.

Redirect Chain Analysis

Medium ▼

What It Checks

Analyzes redirect chains, loops, and excessive redirects that slow down crawling and waste crawler budget.

Why It's Important

Long redirect chains waste crawler budget and can prevent proper indexing. Redirect loops can block crawlers entirely.

Scoring Method

100 for no redirects, 75 for 1 redirect, 50 for 2 redirects, 0 for 3+ or redirect loops.

Excellent: 0 redirects Good: 1-2 redirects Fail: 3+ or loop

How To Fix

Update internal links to point directly to final URLs. Remove unnecessary redirects. Fix redirect loops immediately.

Indexation Checks

Verify your content is properly indexed by search engines

Google Index Status

Critical ▼

What It Checks

Uses the site: search operator via Serper API to determine how many pages from your domain are indexed in Google.

Why It's Important

Google has 70% search market share and many AI systems use Google's data. Being indexed in Google is critical for SEO and significantly increases AI visibility.

Scoring Method

100 points for >100 pages indexed, 85 for >10 pages, 75 for limited indexing (<10), 0 if not indexed at all.

Pass: Site indexed Fail: Not indexed

How To Fix

Submit your site to Google Search Console. Create and submit a sitemap. Ensure robots.txt allows Googlebot. Request indexing for important pages.

Bing Index Status

Critical ▼

What It Checks

Checks whether your site is indexed in Bing search. Recommends manual verification via Bing Webmaster Tools.

Why It's Important

ChatGPT Search uses Bing's index as its primary data source. Being indexed in Bing is critical for appearing in AI-powered search results from OpenAI products.

Scoring Method

100 points for >100 pages indexed, 85 for >10 pages, 75 for limited indexing, 0 if not indexed.

Pass: Site indexed Fail: Not indexed

How To Fix

Submit your site to Bing Webmaster Tools. Create and submit a sitemap. Ensure robots.txt allows Bingbot. Use Bing's URL submission API for faster indexing.

Canonical URL Validation

High ▼

What It Checks

Verifies presence and validity of canonical URL tags. Checks for self-referencing vs cross-domain canonicals and proper formatting.

Why It's Important

Canonical URLs prevent duplicate content issues and tell AI crawlers which version of a page to index and cite. Without them, crawlers may index the wrong URL or split authority.

Scoring Method

100 for valid self-referencing canonical, 75 for cross-domain canonical, 50 for malformed, 0 if missing.

Pass: Valid canonical Warning: Cross-domain Fail: Missing

How To Fix

Add a canonical link tag in your HTML head section pointing to the preferred URL for each page.

Implementation Example

<head>
  <link rel="canonical" href="https://example.com/page-url">
</head>

X-Robots-Tag Headers

Critical ▼

What It Checks

Examines HTTP headers for X-Robots-Tag directives (noindex, nofollow, noarchive, etc.) that control crawler behavior.

Why It's Important

X-Robots-Tag headers can block AI crawlers at the server level, even when content appears accessible. A noindex directive completely prevents indexing.

Scoring Method

100 if no blocking directives, 50 for partial restrictions, 0 for noindex directive.

Pass: No blocking Warning: Some restrictions Fail: noindex present

How To Fix

Review your server configuration and remove blocking X-Robots-Tag directives for pages you want indexed by AI systems.

Security Checks

Security signals that build trust with AI systems

HTTPS Security

Critical ▼

What It Checks

Verifies SSL/TLS certificate implementation and validity, checks for proper HTTPS configuration and HTTP to HTTPS redirects.

Why It's Important

AI systems prioritize secure sources as trust signals. HTTPS is a fundamental trust indicator that influences content credibility and citation likelihood. Non-HTTPS sites may be skipped entirely.

Scoring Method

100 points for valid HTTPS, 95 for HTTPS with proper HTTP redirect, 0 for HTTP only.

Pass: Valid HTTPS Fail: HTTP only

How To Fix

Install an SSL certificate (free from Let's Encrypt), configure your server for HTTPS, and redirect all HTTP traffic to HTTPS.

Mixed Content Detection

Critical ▼

What It Checks

Scans for HTTP resources (images, scripts, styles) loaded on HTTPS pages, which creates security warnings.

Why It's Important

Mixed content degrades security and may cause browsers to block resources. This affects page rendering and can signal poor site maintenance to AI systems.

Scoring Method

100 for no mixed content, 50 for passive mixed content only, 0 for active mixed content.

Pass: No mixed content Warning: Passive only Fail: Active mixed

How To Fix

Update all resource URLs to use HTTPS. Use protocol-relative URLs (//) or full HTTPS URLs. Check for hardcoded HTTP URLs in your templates.

Content Security Policy

Medium ▼

What It Checks

Analyzes Content Security Policy headers and their potential impact on AI crawlers and content extraction.

Why It's Important

Overly restrictive CSP can block legitimate crawlers and prevent content indexing. Balance security with accessibility.

Scoring Method

100 for balanced CSP or none, 50 for restrictive CSP that may block crawlers.

Pass: No blocking CSP Warning: Restrictive CSP

How To Fix

Ensure CSP allows legitimate crawler access. Avoid overly restrictive frame-src or default-src directives. Test CSP changes thoroughly.

Performance Checks

Speed and optimization metrics that affect crawling efficiency

Page Load Speed

High ▼

What It Checks

Measures total page load time and evaluates performance through Google PageSpeed Insights API, including Core Web Vitals (LCP, TBT, CLS).

Why It's Important

Fast-loading pages indicate quality infrastructure and are more likely to be crawled completely. Slow pages may timeout during AI crawling, resulting in incomplete indexing.

Scoring Method

100 for excellent (all vitals good + score ≥90), 85 for good, 70 for acceptable, 50 for poor, 25 for very poor.

Excellent: <1s Good: <2s Pass: <3s Fail: >5s

How To Fix

Optimize images, minify CSS/JS, enable compression, use browser caching, implement lazy loading, and consider using a CDN.

Core Web Vitals

Medium ▼

What It Checks

Evaluates Google's Core Web Vitals: LCP (Largest Contentful Paint), TBT (Total Blocking Time), and CLS (Cumulative Layout Shift).

Why It's Important

Core Web Vitals are performance metrics that indicate user experience quality. AI systems consider these for source credibility and ranking.

Target Thresholds

LCP (Largest Contentful Paint): Good: <2.5s | Needs Improvement: 2.5-4s | Poor: >4s
TBT (Total Blocking Time): Good: <200ms | Needs Improvement: 200-600ms | Poor: >600ms
CLS (Cumulative Layout Shift): Good: <0.1 | Needs Improvement: 0.1-0.25 | Poor: >0.25

How To Fix

Use Google PageSpeed Insights to identify specific issues. Common fixes: preload critical resources, defer non-essential JS, add image dimensions, optimize fonts.

Compression (gzip/brotli)

Medium ▼

What It Checks

Analyzes Content-Encoding response header for compression methods (gzip, br/brotli, deflate).

Why It's Important

Compressed responses load faster, improving crawler efficiency and page speed scores. Brotli typically provides 15-25% better compression than gzip.

Scoring Method

100 for brotli, 90 for gzip, 50 for no compression.

Excellent: brotli Good: gzip Fail: none

How To Fix

Enable gzip or brotli compression in your web server configuration (nginx, Apache, IIS) or through your CDN.

HTTP/2 Support

Low ▼

What It Checks

Detects HTTP protocol version (HTTP/1.1, HTTP/2, HTTP/3) via ALPN negotiation.

Why It's Important

HTTP/2 improves page load speed through multiplexing, header compression, and server push, benefiting crawler efficiency.

Scoring Method

100 for HTTP/2 or HTTP/3, 60 for HTTP/1.1.

Excellent: HTTP/2+ Pass: HTTP/1.1

How To Fix

Enable HTTP/2 on your web server (nginx 1.9.5+, Apache 2.4.17+) or use a CDN that supports it (most do by default).

DNS Prefetch/Preconnect

Low ▼

What It Checks

Looks for resource hints including dns-prefetch, preconnect, preload, and prefetch link elements.

Why It's Important

Resource hints improve page load performance by establishing early connections to critical third-party resources, benefiting both users and crawlers.

Scoring Method

100 for comprehensive hints, 75 for some hints, 50 for none.

Excellent: Multiple hints Good: Some hints Pass: Basic setup

How To Fix

Add preconnect for critical third-party domains, dns-prefetch for other external resources, and preload for critical assets.

Implementation Example

<head>
  <!-- Preconnect to critical origins -->
  <link rel="preconnect" href="https://fonts.googleapis.com">
  <link rel="preconnect" href="https://cdn.example.com" crossorigin>

  <!-- DNS prefetch for other domains -->
  <link rel="dns-prefetch" href="https://analytics.example.com">

  <!-- Preload critical resources -->
  <link rel="preload" href="/fonts/main.woff2" as="font" type="font/woff2" crossorigin>
</head>

Server Response Headers

Low ▼

What It Checks

Analyzes Cache-Control, ETag, Last-Modified, and other caching headers for proper configuration.

Why It's Important

Proper caching headers improve performance and help crawlers efficiently re-crawl content by knowing when pages have changed.

Scoring Method

100 for comprehensive caching setup (Cache-Control + ETag), scaled down for missing headers.

Excellent: Cache-Control + ETag Good: Cache-Control Pass: Any caching

How To Fix

Configure Cache-Control headers with appropriate max-age values. Add ETag headers for validation. Enable Last-Modified headers.

Compatibility Checks

Device and language compatibility for broader reach

Mobile Responsiveness

High ▼

What It Checks

Verifies the viewport meta tag and presence of responsive CSS media queries for mobile-friendly design.

Why It's Important

Mobile-friendly content is prioritized by search engines and AI systems as it indicates modern, well-maintained content. Google uses mobile-first indexing.

Scoring Method

100 for viewport + media queries, -50 for missing viewport, -25 for no media queries.

Pass: Viewport + media queries Warning: Viewport only Fail: Neither

How To Fix

Add the viewport meta tag and implement responsive CSS with media queries.

Implementation Example

<!-- Add to <head> -->
<meta name="viewport" content="width=device-width, initial-scale=1.0">

<!-- CSS media queries -->
<style>
@media (max-width: 768px) {
  .container { padding: 10px; }
  .sidebar { display: none; }
}
</style>

Hreflang Tags

Medium ▼

What It Checks

Analyzes hreflang attribute implementation for multilingual and multi-regional content targeting.

Why It's Important

Hreflang helps AI systems serve the correct language version of your content to users in different regions, improving relevance and reducing duplicate content issues.

Scoring Method

100 for properly implemented hreflang, 50 for partial implementation, 0 if missing on multilingual sites.

Pass: Valid hreflang Warning: Partial Fail: Missing on multilingual

How To Fix

Add hreflang link elements for each language version, including a self-referencing tag and x-default for fallback.

Implementation Example

<head>
  <link rel="alternate" hreflang="en" href="https://example.com/page">
  <link rel="alternate" hreflang="es" href="https://example.com/es/page">
  <link rel="alternate" hreflang="fr" href="https://example.com/fr/page">
  <link rel="alternate" hreflang="x-default" href="https://example.com/page">
</head>

AI-Specific Technical Checks

Technical configurations specifically for AI optimization

llms.txt File

High ▼

What It Checks

Verifies the presence and format of /llms.txt file, which provides explicit AI guidance about your site similar to robots.txt but for content interpretation.

Why It's Important

llms.txt provides explicit instructions to AI systems about your content, helping them understand what your site is about, key topics, and how to interpret your content correctly.

Scoring Method

100 for well-formatted file with sections, 50 for partial content, 0 if missing.

Pass: Complete llms.txt Warning: Partial Fail: Missing

How To Fix

Create a /llms.txt file at your domain root with structured information about your site, key topics, and content guidelines.

Implementation Example

# llms.txt - AI Guidance for Example.com

# Site Information
Title: Example Company
URL: https://example.com
Description: Leading provider of innovative solutions in the tech industry.

# Key Topics
- Product development and innovation
- Industry best practices
- Technical tutorials and guides
- Company news and updates

# Content Guidelines
- All content is fact-checked and regularly updated
- Technical content reviewed by industry experts
- Product information accurate as of publication date

# Important Pages
/about - Company overview and mission
/products - Product catalog and specifications
/blog - Technical articles and industry insights
/contact - Contact information and support

# Update Frequency
Blog: Weekly
Products: Monthly
Documentation: As needed

AI Meta Tags

Medium ▼

What It Checks

Looks for AI-specific meta tags including ai-content-type, ai-description, ai-keywords, chatgpt-guidance, and claude-guidance.

Why It's Important

These emerging meta tags provide direct signals to AI systems about content type and handling preferences, potentially improving how AI interprets and presents your content.

Scoring Method

25 points per AI meta tag found (max 100 with 4+ tags).

Excellent: 4+ tags Good: 2-3 tags Pass: 1 tag Fail: 0 tags

How To Fix

Add AI-specific meta tags to your HTML head section based on your content type and AI interaction preferences.

Implementation Example

<head>
  <meta name="ai-content-type" content="article">
  <meta name="ai-description" content="Comprehensive guide to AI Engine Optimization">
  <meta name="ai-keywords" content="AEO, AI optimization, content strategy">
  <meta name="chatgpt-guidance" content="This is a technical tutorial with code examples">
  <meta name="claude-guidance" content="Focus on the step-by-step instructions">
</head>

OpenGraph Tags

Medium ▼

What It Checks

Verifies presence of essential OpenGraph meta tags: og:title, og:description, og:image, og:url, and og:type.

Why It's Important

OpenGraph tags define how content appears when shared on social platforms and help AI systems understand content metadata for better categorization and presentation.

Scoring Method

20 points per required tag present (5 tags = 100 points).

Excellent: All 5 tags Good: 4 tags Pass: 3 tags Fail: <3 tags

How To Fix

Add all 5 essential OpenGraph meta tags to your page's head section.

Implementation Example

<head>
  <meta property="og:title" content="Your Page Title">
  <meta property="og:description" content="A compelling description of your page content">
  <meta property="og:image" content="https://example.com/image.jpg">
  <meta property="og:url" content="https://example.com/page">
  <meta property="og:type" content="article">
</head>

Twitter Cards

Medium ▼

What It Checks

Verifies presence of Twitter Card meta tags: twitter:card, twitter:title, twitter:description, and twitter:image.

Why It's Important

Twitter Cards optimize content display on Twitter/X and provide additional metadata signals that AI systems can use for content understanding.

Scoring Method

25 points per tag present (max 100 with all 4 tags).

Excellent: All 4 tags Good: 3 tags Pass: 2 tags Fail: <2 tags

How To Fix

Add Twitter Card meta tags to your page's head section. Use 'summary_large_image' for best visual presentation.

Implementation Example

<head>
  <meta name="twitter:card" content="summary_large_image">
  <meta name="twitter:title" content="Your Page Title">
  <meta name="twitter:description" content="A compelling description">
  <meta name="twitter:image" content="https://example.com/image.jpg">
</head>