Technical & Foundational Health
Technical checks evaluate your website's infrastructure, security, crawlability, and performance. These foundational elements determine whether AI systems can access and properly index your content.
Critical checks first: Focus on fixing critical issues (robots.txt, HTTPS, crawlability, noindex settings) before addressing other technical optimizations.
How these checks count: Each check is weighted by how well the evidence supports it. A check marked Unproven is shown in your report but not scored. A check marked N/A or "could not test" is left out of the score rather than counted as a pass or a failure. See the scoring system for the details.
Crawlability Checks
Ensure AI crawlers can access your content
robots.txt and AI Crawler Access
Critical
What It Checks
Two checks read your robots.txt. robots.txt Validation checks that the file exists, can be read, has User-agent lines and names your sitemap. AI Bot Permissions reads the rules the way crawlers do (RFC 9309) and judges each AI crawler on the page that was scanned, grouped by purpose: search (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, Googlebot), user fetch (ChatGPT-User, Perplexity-User, Claude-User) and training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent). Crawler names as of 2026-10-07.
Why It's Important
Search and user-fetch crawlers decide whether AI search and assistants can open a page, so blocking one on a page you want found is a real problem. Training crawlers are a separate decision: blocking GPTBot or Google-Extended is your policy choice and does not hide the page from AI search. A crawler with no group of its own follows your
User-agent: * rules, so you do not need an Allow line for every bot.
Scoring Method
robots.txt Validation: 100 for a readable file with User-agent lines and a Sitemap line, 80 without the Sitemap line, 50 without User-agent lines, 0 when there is no file. AI Bot Permissions: 100 when no search or user-fetch crawler is blocked on the page (training blocks are fine). It fails with 0 when every search crawler is blocked, 25 when some are, and 50 when only user-fetch crawlers are blocked. If robots.txt cannot be read (a firewall refusal, a server error, a redirect or an HTML page instead of rules) the check is not measured and left out of the score.
Pass: no search or user-fetch crawler is blocked on the page
Not measured: robots.txt could not be read
Fail: a search or user-fetch crawler is blocked on the page
How To Fix
Give a blocked search or user-fetch crawler its own group with
Allow: /, or remove the rule that blocks it, and repeat your private-path Disallow lines in that group (a named group does not inherit the * rules). Decide training access separately. Put the file at the root of your domain (e.g., example.com/robots.txt).
Implementation Example
# robots.txt - allow AI search, decline training (example policy) User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / Disallow: /admin/ User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended Disallow: / User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Sitemap Validation
High
What It Checks
Verifies the existence and validity of your sitemap (at the standard locations or declared in robots.txt): a real XML or plain-text sitemap that lists URLs. priority and changefreq are not scored, because Google ignores them.
Why It's Important
A sitemap is the list of pages you tell search engines you want found. It helps discovery; it does not guarantee that a page is indexed.
Scoring Method
100 points for a valid sitemap that lists URLs. 20 when it answers but lists no URLs. 0 if missing; partial credit (50 or 30) when it exists or may exist but our request is blocked.
Pass: Valid sitemap exists
Fail: No sitemap or invalid
How To Fix
Create an XML sitemap at /sitemap.xml listing the canonical address of every page you want found, with a lastmod that changes only when the content does.
Implementation Example
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2024-01-15</lastmod>
</url>
<url>
<loc>https://example.com/about</loc>
<lastmod>2024-01-10</lastmod>
</url>
</urlset>
Sitemap Membership
Medium
What It Checks
Looks the scanned page up in the sitemaps we read: the sitemap at a standard location or declared in robots.txt, its child sitemaps (up to 10) and the other sitemaps robots.txt declares, up to 50,000 URLs in all. The page's own address, the address it lands on after redirects and its canonical all count.
Why It's Important
Your sitemap is the list of pages you tell search engines you want found. A page missing from it is harder to discover, and a sitemap that leaves out real pages is not a truthful list.
Scoring Method
100 when the page is listed. 0 when the sitemaps were read in full and the page is not in them.
Pass: the page or its canonical is listed
Fail: sitemaps read in full, page not listed
N/A: no sitemap, an empty or unreadable sitemap, a sitemap only partly read (could not test), or a noindex page
How To Fix
Add the page's canonical address to your sitemap. Better, generate the sitemap from the same list of canonical, indexable pages your site publishes, so new pages are added and retired ones removed automatically.
Site Crawlability
Critical
What It Checks
Tests basic accessibility of your site including HTTP response codes, server availability, and detection of bot protection systems (Cloudflare, Akamai, etc.).
Why It's Important
If AI crawlers can't access your site, your content won't be indexed or cited. Bot protection that blocks legitimate crawlers prevents AI visibility entirely.
Scoring Method
100 points for 200 OK response. 30 points for bot protection detected (403). 0 for server errors (5xx).
Pass: 2xx responses
Fail: 4xx/5xx errors
How To Fix
Ensure server uptime, whitelist known AI bot user agents in your WAF/CDN, fix broken pages, and handle errors gracefully.
Redirect Chain Analysis
Medium
What It Checks
Analyzes redirect chains, loops, and excessive redirects that slow down crawling and waste crawler budget.
Why It's Important
Long redirect chains waste crawler budget and can prevent proper indexing. Redirect loops can block crawlers entirely.
Scoring Method
100 for no redirects, 75 for 1 redirect, 50 for 2 redirects, 0 for 3+ or redirect loops.
Excellent: 0 redirects
Good: 1-2 redirects
Fail: 3+ or loop
How To Fix
Update internal links to point directly to final URLs. Remove unnecessary redirects. Fix redirect loops immediately.
Indexation Checks
Verify your content is properly indexed by search engines
Google Index Status
Critical
What It Checks
Uses the site: search operator via Serper API to determine how many pages from your domain are indexed in Google.
Why It's Important
Google has 70% search market share and many AI systems use Google's data. Being indexed in Google is critical for SEO and significantly increases AI visibility.
Scoring Method
100 points for >100 pages indexed, 85 for >10 pages, 75 for limited indexing (<10), 0 if not indexed at all.
Pass: Site indexed
Fail: Not indexed
How To Fix
Submit your site to Google Search Console. Create and submit a sitemap. Ensure robots.txt allows Googlebot. Request indexing for important pages.
Bing Index Status
Critical
What It Checks
Checks whether your site is indexed in Bing search. Recommends manual verification via Bing Webmaster Tools.
Why It's Important
ChatGPT Search uses Bing's index as its primary data source. Being indexed in Bing is critical for appearing in AI-powered search results from OpenAI products.
Scoring Method
100 points for >100 pages indexed, 85 for >10 pages, 75 for limited indexing, 0 if not indexed.
Pass: Site indexed
Fail: Not indexed
How To Fix
Submit your site to Bing Webmaster Tools. Create and submit a sitemap. Ensure robots.txt allows Bingbot. Use Bing's URL submission API for faster indexing.
Canonical URL Validation
High
What It Checks
Verifies presence and validity of canonical URL tags. Checks for self-referencing vs cross-domain canonicals and proper formatting.
Why It's Important
A canonical URL tells search engines which address to keep when the same content is reachable at several. Without one, they pick for themselves and may keep the wrong address.
Scoring Method
Starts at 100, minus 40 for more than one canonical tag, 20 for a canonical on the same site that is another page, 20 for a relative address, 20 for a different protocol. Passes at 70 or more with a single tag; 0 if missing. A cross-domain canonical is allowed. Whether the address works is checked by Canonical Target below.
Pass: Valid canonical
Warning: Relative address or other page
Fail: Missing or more than one tag
How To Fix
Add a canonical link tag in your HTML head section pointing to the preferred URL for each page.
Implementation Example
<head> <link rel="canonical" href="https://example.com/page-url"> </head>
Canonical Target
High
What It Checks
Opens the canonical URL your page declares (its canonical link tag, else the Link header) when it names another page, and checks that it answers HTTP 200 directly, without a redirect, an error or a noindex directive. A canonical that is the page itself needs no test.
Why It's Important
The canonical tells search engines which address to keep. If that address redirects, is broken or is marked noindex, you are pointing them at a page they cannot use, and they may ignore your preference or drop the page.
Scoring Method
100 when the canonical opens directly with 200 and may be indexed. 0 when it redirects, returns an error (Critical for 404 and 410), is marked noindex or is not a web address.
Pass: the canonical opens directly (HTTP 200) and may be indexed
Fail: redirect, error status or noindex
N/A: no canonical, the canonical is the page itself, or we could not test (timeout, refusal, bot challenge)
How To Fix
Make the canonical the full, final https:// address of a live page, usually the page's own address. Find where the value comes from (page setting, template, CMS default or plugin), fix it there, clear caches and re-scan.
Soft 404
High
What It Checks
Looks for a page that answers with a success status but shows a "page not found" message: a title or main heading such as "Page not found", "404 | Shop" or "Pagina non trovata" on a short page, or a page-specific phrase such as "this page does not exist" on an almost empty page. English and Italian wording is recognised. A bare "404" elsewhere ("Peugeot 404 for sale") does not count.
Why It's Important
Search engines treat these pages as error pages, or index the error text as if it were your content. Honest status codes keep missing pages out of results and real pages in.
Scoring Method
100 when the page's content matches its success status. 0 when it says it was not found. Not applicable when the page did not answer with a 2xx status.
Pass: no "not found" message on a page that answered 200
Fail: "not found" message on a page that answered 200
N/A: the page did not answer with a success status
How To Fix
If the page is gone, make the server answer 404 (or 410 when removed for good). If it should exist, restore its content, or redirect it to a real equivalent page, not to the home page.
X-Robots-Tag Headers
Critical
What It Checks
Examines HTTP headers for X-Robots-Tag directives (noindex, nofollow, noarchive, etc.) that control crawler behavior.
Why It's Important
An X-Robots-Tag header can keep a page out of search results at the server level, even when the page looks fine in a browser. A noindex directive prevents indexing. Robots meta tags and the other snippet controls are scored by Snippet Controls below, so the same noindex is not counted twice.
Scoring Method
100 if no blocking directives, 50 when the header says nofollow, 30 for a noindex (or none) directive.
Pass: No blocking
Warning: Some restrictions
Fail: noindex present
How To Fix
Review your server configuration and remove blocking X-Robots-Tag directives for pages you want found in search and AI answers.
Snippet Controls
Critical
What It Checks
Reads the robots, googlebot and bingbot meta tags and the X-Robots-Tag header for noindex, nosnippet, max-snippet:0 and none, and measures how much of the text is wrapped in data-nosnippet. A noindex sent only in the X-Robots-Tag header is scored by X-Robots-Tag, not here.
Why It's Important
noindex keeps the page out of search results. nosnippet and max-snippet:0 stop search engines from showing its text, and Google applies them to its AI features too. data-nosnippet hides only the passages you mark; that is your choice.
Scoring Method
100 when none of those directives is set. 0 when one is (Critical for noindex or none, High for nosnippet or max-snippet:0). data-nosnippet never fails the check.
Pass: no noindex, nosnippet, max-snippet:0 or none
Fail: one of them is set for Google or Bing
How To Fix
Remove the directive from the page template and from server or CDN headers on pages that should appear in search and AI answers. Keep data-nosnippet for passages you really want kept out of snippets.
Security Checks
Security signals that build trust with AI systems
HTTPS Security
Critical
What It Checks
Verifies SSL/TLS certificate implementation and validity, checks for proper HTTPS configuration and HTTP to HTTPS redirects.
Why It's Important
AI systems prioritize secure sources as trust signals. HTTPS is a fundamental trust indicator that influences content credibility and citation likelihood. Non-HTTPS sites may be skipped entirely.
Scoring Method
100 points for valid HTTPS, 95 for HTTPS with proper HTTP redirect, 0 for HTTP only.
Pass: Valid HTTPS
Fail: HTTP only
How To Fix
Install an SSL certificate (free from Let's Encrypt), configure your server for HTTPS, and redirect all HTTP traffic to HTTPS.
Mixed Content Detection
Critical
What It Checks
Scans for HTTP resources (images, scripts, styles) loaded on HTTPS pages, which creates security warnings.
Why It's Important
Mixed content degrades security and may cause browsers to block resources. This affects page rendering and can signal poor site maintenance to AI systems.
Scoring Method
100 for no mixed content, 50 for passive mixed content only, 0 for active mixed content.
Pass: No mixed content
Warning: Passive only
Fail: Active mixed
How To Fix
Update all resource URLs to use HTTPS. Use protocol-relative URLs (//) or full HTTPS URLs. Check for hardcoded HTTP URLs in your templates.
Content Security Policy
Retired
Status
Retired as a scored technical check: it has no effect on your score. The Security tab of a report still audits your Content Security Policy header for information.
Performance Checks
Speed and optimization metrics that affect crawling efficiency
Page Load Speed
High
What It Checks
Measures total page load time and evaluates performance through Google PageSpeed Insights API, including Core Web Vitals (LCP, TBT, CLS).
Why It's Important
Fast-loading pages indicate quality infrastructure and are more likely to be crawled completely. Slow pages may timeout during AI crawling, resulting in incomplete indexing.
Scoring Method
100 for excellent (all vitals good + score ≥90), 85 for good, 70 for acceptable, 50 for poor, 25 for very poor.
Excellent: <1s
Good: <2s
Pass: <3s
Fail: >5s
How To Fix
Optimize images, minify CSS/JS, enable compression, use browser caching, implement lazy loading, and consider using a CDN.
Core Web Vitals
Medium
What It Checks
Evaluates Google's Core Web Vitals: LCP (Largest Contentful Paint), TBT (Total Blocking Time), and CLS (Cumulative Layout Shift).
Why It's Important
Core Web Vitals are performance metrics that indicate user experience quality. AI systems consider these for source credibility and ranking.
Target Thresholds
- LCP (Largest Contentful Paint): Good: <2.5s | Needs Improvement: 2.5-4s | Poor: >4s
- TBT (Total Blocking Time): Good: <200ms | Needs Improvement: 200-600ms | Poor: >600ms
- CLS (Cumulative Layout Shift): Good: <0.1 | Needs Improvement: 0.1-0.25 | Poor: >0.25
How To Fix
Use Google PageSpeed Insights to identify specific issues. Common fixes: preload critical resources, defer non-essential JS, add image dimensions, optimize fonts.
Compression (gzip/brotli)
Medium
What It Checks
Analyzes Content-Encoding response header for compression methods (gzip, br/brotli, deflate).
Why It's Important
Compressed responses load faster, improving crawler efficiency and page speed scores. Brotli typically provides 15-25% better compression than gzip.
Scoring Method
100 for brotli, 90 for gzip, 50 for no compression.
Excellent: brotli
Good: gzip
Fail: none
How To Fix
Enable gzip or brotli compression in your web server configuration (nginx, Apache, IIS) or through your CDN.
HTTP/2 Support
Low
What It Checks
Detects HTTP protocol version (HTTP/1.1, HTTP/2, HTTP/3) via ALPN negotiation.
Why It's Important
HTTP/2 improves page load speed through multiplexing, header compression, and server push, benefiting crawler efficiency.
Scoring Method
100 for HTTP/2 or HTTP/3, 60 for HTTP/1.1.
Excellent: HTTP/2+
Pass: HTTP/1.1
How To Fix
Enable HTTP/2 on your web server (nginx 1.9.5+, Apache 2.4.17+) or use a CDN that supports it (most do by default).
DNS Prefetch/Preconnect
Low
What It Checks
Looks for resource hints including dns-prefetch, preconnect, preload, and prefetch link elements.
Why It's Important
Resource hints improve page load performance by establishing early connections to critical third-party resources, benefiting both users and crawlers.
Scoring Method
100 for comprehensive hints, 75 for some hints, 50 for none.
Excellent: Multiple hints
Good: Some hints
Pass: Basic setup
How To Fix
Add preconnect for critical third-party domains, dns-prefetch for other external resources, and preload for critical assets.
Implementation Example
<head> <!-- Preconnect to critical origins --> <link rel="preconnect" href="https://fonts.googleapis.com"> <link rel="preconnect" href="https://cdn.example.com" crossorigin> <!-- DNS prefetch for other domains --> <link rel="dns-prefetch" href="https://analytics.example.com"> <!-- Preload critical resources --> <link rel="preload" href="/fonts/main.woff2" as="font" type="font/woff2" crossorigin> </head>
Server Response Headers
Low
What It Checks
Analyzes Cache-Control, ETag, Last-Modified, and other caching headers for proper configuration.
Why It's Important
Proper caching headers improve performance and help crawlers efficiently re-crawl content by knowing when pages have changed.
Scoring Method
100 for comprehensive caching setup (Cache-Control + ETag), scaled down for missing headers.
Excellent: Cache-Control + ETag
Good: Cache-Control
Pass: Any caching
How To Fix
Configure Cache-Control headers with appropriate max-age values. Add ETag headers for validation. Enable Last-Modified headers.
Compatibility Checks
Device and language compatibility for broader reach
Mobile Responsiveness
High
What It Checks
Verifies the viewport meta tag and presence of responsive CSS media queries for mobile-friendly design.
Why It's Important
Mobile-friendly content is prioritized by search engines and AI systems as it indicates modern, well-maintained content. Google uses mobile-first indexing.
Scoring Method
100 for viewport + media queries, -50 for missing viewport, -25 for no media queries.
Pass: Viewport + media queries
Warning: Viewport only
Fail: Neither
How To Fix
Add the viewport meta tag and implement responsive CSS with media queries.
Implementation Example
<!-- Add to <head> -->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- CSS media queries -->
<style>
@media (max-width: 768px) {
.container { padding: 10px; }
.sidebar { display: none; }
}
</style>
Hreflang Tags
Medium
What It Checks
Analyzes hreflang attribute implementation for multilingual and multi-regional content targeting.
Why It's Important
Hreflang helps AI systems serve the correct language version of your content to users in different regions, improving relevance and reducing duplicate content issues.
Scoring Method
100 for properly implemented hreflang, 50 for partial implementation, 0 if missing on multilingual sites.
Pass: Valid hreflang
Warning: Partial
Fail: Missing on multilingual
How To Fix
Add hreflang link elements for each language version, including a self-referencing tag and x-default for fallback.
Implementation Example
<head> <link rel="alternate" hreflang="en" href="https://example.com/page"> <link rel="alternate" hreflang="es" href="https://example.com/es/page"> <link rel="alternate" hreflang="fr" href="https://example.com/fr/page"> <link rel="alternate" hreflang="x-default" href="https://example.com/page"> </head>
AI-Specific Technical Checks
Technical configurations specifically for AI optimization
llms.txt File
Unproven · not scored
What It Checks
Looks for a plain-text /llms.txt (or /llm.txt) file at your domain root and rates how complete it looks.
Why It's Important
llms.txt is an optional summary file. Google does not use it and no major AI engine is shown to, so a missing file is not a problem. The check is shown for information and does not count in your score.
Scoring Method
25 points each for a title header, a description and URLs, 15 for two or more sections and 10 for more than 500 characters. Passes at 50. Not scored: unproven checks carry no weight.
Pass: 50 or more
Fail: missing or below 50
How To Fix
If you want one, publish a plain-text /llms.txt with a title, a short description and links to your key pages. Skipping it costs you nothing in the score.
Implementation Example
# Example Company > Leading provider of innovative solutions in the tech industry. ## Important pages - [About](https://example.com/about): Company overview and mission - [Products](https://example.com/products): Product catalog and specifications - [Blog](https://example.com/blog): Technical articles and industry insights ## Contact - Email: contact@example.com
AI Meta Tags
Retired
Status
Retired. This check no longer runs, so it does not appear in reports and has no effect on your score.
OpenGraph Tags
Medium
What It Checks
Verifies presence of essential OpenGraph meta tags: og:title, og:description, og:image, og:url, and og:type.
Why It's Important
OpenGraph tags define how content appears when shared on social platforms and help AI systems understand content metadata for better categorization and presentation.
Scoring Method
20 points per required tag present (5 tags = 100 points).
Excellent: All 5 tags
Good: 4 tags
Pass: 3 tags
Fail: <3 tags
How To Fix
Add all 5 essential OpenGraph meta tags to your page's head section.
Implementation Example
<head> <meta property="og:title" content="Your Page Title"> <meta property="og:description" content="A compelling description of your page content"> <meta property="og:image" content="https://example.com/image.jpg"> <meta property="og:url" content="https://example.com/page"> <meta property="og:type" content="article"> </head>
Twitter Cards
Medium
What It Checks
Verifies presence of Twitter Card meta tags: twitter:card, twitter:title, twitter:description, and twitter:image.
Why It's Important
Twitter Cards optimize content display on Twitter/X and provide additional metadata signals that AI systems can use for content understanding.
Scoring Method
25 points per tag present (max 100 with all 4 tags).
Excellent: All 4 tags
Good: 3 tags
Pass: 2 tags
Fail: <2 tags
How To Fix
Add Twitter Card meta tags to your page's head section. Use 'summary_large_image' for best visual presentation.
Implementation Example
<head> <meta name="twitter:card" content="summary_large_image"> <meta name="twitter:title" content="Your Page Title"> <meta name="twitter:description" content="A compelling description"> <meta name="twitter:image" content="https://example.com/image.jpg"> </head>