---
title: "The AI Visibility Checklist: 21 Fixes in the Order Bots Fetch Your Site"
description: "Run 21 technical SEO checks in fetch order, from robots.txt and edge access to answer pages and freshness. Stop at the first failure before polishing schema."
url: "https://groas.com/post/the-ai-visibility-technical-checklist-21"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2f6bd5c0-8971-4b3e-b629-6b39020c2b09.png"
published: "2026-10-16T05:38:48.601Z"
modified: "2026-10-16T05:38:48.664Z"
---

October 16, 2026 · 9 min read

# The AI Visibility Checklist: 21 Fixes in the Order Bots Fetch Your Site

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Tiny courier stopped at a locked first gate on a model road, while far away workers gild a shop sign nobody can reach—polish is useless without access.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2f6bd5c0-8971-4b3e-b629-6b39020c2b09.png)

In this article

1. [Tier 1: Let the bot through the file, firewall and server](#tier-1-let-the-bot-through-the-file-firewall-and-server)
2. [Tier 2: Make the answer page discoverable](#tier-2-make-the-answer-page-discoverable)
3. [Tier 3: Put the answer in HTML the bot can read](#tier-3-put-the-answer-in-html-the-bot-can-read)
4. [Tier 4: Give it a passage worth citing](#tier-4-give-it-a-passage-worth-citing)
5. [Tier 5: Make the source unmistakable](#tier-5-make-the-source-unmistakable)
6. [Tier 6: Update the page the bot already knows](#tier-6-update-the-page-the-bot-already-knows)
7. [Before you buy another visibility dashboard](#before-you-buy-another-visibility-dashboard)

Most AI visibility checklists start with schema because audit tools put schema on the report. **I start with robots.txt because bots request access before they can judge a page.** On our own site, robots.txt drew **3,867 verified AI hits and 1,222 live-answer fetches**, more than the homepage in the same 30-day window [I logged in our crawler teardown](https://groas.com/post/we-logged-30-days-of-ai-bots-on-our-own). A mistake there costs more than a schema gap: no fetch means no answer to lift, entity to resolve or update date to notice. Run these 21 checks in order. Stop at the first failure.

## Tier 1: Let the bot through the file, firewall and server

![AI bot stopped at a robots.txt gate before reaching website pages](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/90d8bd1d-7f23-4b42-94e8-6858f7305b5b.png)

1. **Allow `OAI-SearchBot`.** Pass/fail: its robots.txt rules do not block the pages you want in ChatGPT search answers. [OpenAI assigns its crawlers different jobs](https://developers.openai.com/api/docs/bots): `OAI-SearchBot` serves search, while `GPTBot` is for training. You can allow search without allowing training. Check the actual rule group rather than assuming one decision covers both; if you change it, allow roughly 24 hours for the robots.txt change to propagate.
2. **Allow `PerplexityBot`; handle `Perplexity-User` separately.** Pass/fail: the indexing bot is permitted and can get through your server or WAF. [PerplexityBot builds the index used for surfacing pages](https://docs.perplexity.ai/guides/bots); it is not the training bot. [Perplexity-User generally ignores robots.txt](https://docs.perplexity.ai/guides/bots) because a person requested the page. If you need to control those user-triggered fetches, do it at the server or WAF, not with a rule they may not follow. Check Perplexity’s published IP guidance when configuring access.
3. **Allow `ClaudeBot`; keep `Google-Extended` in its own lane.** Pass/fail: no broad `Disallow: /` hides under an AI-labeled rule you meant to use more narrowly. [`Google-Extended` is a robots.txt token for Gemini training choice, not a separate crawler](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers); Googlebot still handles ordinary crawling. Set `Google-Extended` according to your training preference without confusing it with Search or AI Overviews access. Keep Googlebot allowed, and use [the AI crawler swipe file](https://groas.com/post/the-ai-crawler-swipe-file-copy-paste-rob) rather than improvising rule blocks.
4. **Check the edge before you blame robots.txt.** Pass/fail: permitted bot traffic gets a `200`, not a `403` or a challenge, when it requests the homepage. A user-agent request is a useful first check, but read your edge and server logs too; typing a bot name into a header does not make your request that bot. [Cloudflare announced AI-crawler blocking changes on July 1, 2025](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/), and [Perplexity publishes WAF whitelisting guidance](https://docs.perplexity.ai/guides/bots). Check Cloudflare’s Security > Bots and AI Crawl Control, or AWS WAF’s managed bot rules and IP sets. A clean file cannot overrule a blocking firewall.
5. **Get money pages to `200` without a detour.** Pass/fail: pricing, product and contact pages return `200` in fewer than two redirects, without an interstitial or login wall blocking the fetch. [ChatGPT-User is user-triggered, so robots.txt rules may not apply](https://developers.openai.com/api/docs/bots). That makes the response from your server the check that matters. Strip redirect chains, remove the country-splash redirect for bot traffic and serve the page you want read.

## Tier 2: Make the answer page discoverable

6. **Keep `sitemap.xml` small, fresh and honest.** Pass/fail: it returns `200`, contains only indexable `200` URLs and reports a `lastmod` date that matches a real, visible update. Drop redirects, noindex pages and PDFs you do not want cited. Regenerate the file when you publish. A sitemap full of destinations that do not resolve cleanly sends the bot back into the access problems you just fixed.
7. **Link answer pages from pages bots already fetch.** Pass/fail: every page answering a buyer question has a crawlable `<a href>` from the homepage, pricing page, main category or top blog index. Use a contextual anchor that says what question the destination answers, not a pile of links in the footer. Access does not guarantee discovery: the bot still needs a route from a page it knows to the page you want cited.
8. **Find the orphans before editing their copy.** Pass/fail: check `site:yourdomain.com` and your server logs for each target URL, including whether it was fetched in the last 30 days. A missing log hit in that window is a reason to investigate, not proof that the page can never appear. Add an internal link, put the URL in the sitemap and request indexing in Search Console once. Do not work through items 9 to 21 on an orphan while its route into the site remains broken.

## Tier 3: Put the answer in HTML the bot can read

9. **Server-render the main answer.** Pass/fail: disable JavaScript, reload and read the answer in the page’s raw HTML. Server-render the article body, pricing facts and comparison tables; a client-side React shell that delivers the text only after JavaScript runs is a bad bet. [The crawler analysis discussed here found a JavaScript-rendering gap among major AI crawlers](https://www.searchenginejournal.com/ask-an-seo-can-ai-systems-llms-render-javascript-to-read-hidden-content/559724/). Test _View Source_, not just _Inspect Element_: the latter can show content your browser added after the initial response.
10. **Put key facts in selectable text, not only images.** Pass/fail: every number or comparison you want quoted appears in HTML text. Keep the chart or screenshot for people, then write its important facts underneath it. Alt text is useful, but it is not a replacement for the answer itself. [Content that needs JavaScript to reveal, including some tabs and accordions, poses the same reading problem](https://www.searchenginejournal.com/ask-an-seo-can-ai-systems-llms-render-javascript-to-read-hidden-content/559724/): if the fact is missing from the initial page, do not expect the bot to retrieve it.
11. **Render tab and accordion text on first load.** Pass/fail: _View Source_ contains the full text of every panel you need a bot to read. Put the panels in the initial HTML, use JavaScript only to collapse them visually and give sections anchor links where useful. A polished interactive tab is not helping if the answer exists only after a click. Fix the empty source before you rewrite the copy inside it.

## Tier 4: Give it a passage worth citing

12. **Lead target sections with a direct answer.** Pass/fail: the first sentence after the question heading answers that question without setup. Aim for a self-contained passage of roughly 40 to 60 words, then put the background underneath. If readers need the previous paragraph to decode the answer, the passage is harder to lift on its own. Audit tools may send you to schema first; give the bot a usable sentence first.
13. **Give each buyer intent one clear URL.** Pass/fail: a `site:yourdomain.com` search for the question shows one obvious answer page, not three near-duplicates competing to say the same thing. Merge overlapping posts, redirect the ones you retire with a 301 and keep one live URL for that question. Duplicates split the attention of a bot trying to choose what to fetch and cite. This is consolidation, not permission to bury different buyer questions on a single page.
14. **Use real lists and tables for enumerated facts.** Pass/fail: steps, prices, pros and cons, and comparisons use native `<ol>`, `<ul>` or `<table>` HTML rather than styled divs or screenshots. Rewrite a comparison as a three-column table with the verdict in its cells when that is the clearest shape. Structured HTML keeps the relationships between facts visible and makes a passage easier to carry into an answer. Do not make the bot reconstruct a table from a picture.
15. **Put the claim on the landing page you want cited.** Pass/fail: its definition, price or step appears in text there, not only in a linked PDF or video. Monitoring software may show that the landing page was skipped; it will not move the missing sentence onto it. Make the indexable page answer the question directly. If a person must click twice to find the fact, you have given the bot a longer route than it needs.

## Tier 5: Make the source unmistakable

![Small business storefront with mismatched name signs confusing a robot reader](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/9af7849c-d89a-4018-ac92-b93aa98a0487.png)

16. **Use one consistent `Organization` identity.** Pass/fail: the homepage schema, footer, contact page and Google Business Profile use the same business name. Keep address and phone formats consistent, and keep `sameAs` links aligned with the profiles you claim. I used to tell clients schema was decoration. I was wrong about its usefulness here: it gives the bot another way to connect the answer to the business behind it. Do not make that connection harder with conflicting names.
17. **Give each author a real page.** Pass/fail: each byline links to an `/author/name` page with a photo, role, three prior posts and an external credential. Replace “Admin” bylines and bios written like keyword paragraphs. On local and B2B pages, connect the author to the business rather than a freelancer marketplace. The point is to make attribution legible, not to build an elaborate author page while the article body still arrives as an empty shell.
18. **Link to the entities you claim.** Pass/fail: your about page links to the review profiles, certifications and suppliers it mentions, using matching names. If you say you install Trane systems in Austin, link Trane and your license page. Give the bot a clear route between the claim and the entity behind it. One coherent identity is more useful than five half-filled profiles that disagree about who you are.

## Tier 6: Update the page the bot already knows

19. **Match the visible update date to `lastmod`.** Pass/fail: the date a person sees agrees with the date in the sitemap. Put `dateModified` and a rendered date near the H1, then change them when the page actually changes. Access and rendering decide whether the page can be read at all; freshness matters only after that. In [the Ahrefs 17M-citation analysis](https://generatemore.ai/), AI Overviews cited pages about 16 days older on average than organic results, while ChatGPT skewed fresher. Do not treat a fresh timestamp as a substitute for a reachable answer.
20. **Add a short changelog to buyer pages.** Pass/fail: pricing, comparison and how-to pages say what changed and when. Two plain lines will do; if a plan or limit changed, name the change rather than merely stamping today’s date on the page. Readers checking a consequential fact should not have to guess whether the copy or just the timestamp is new. Keep the note close enough to the answer that it explains the update.
21. **Refresh the winner; do not publish a rival.** Pass/fail: one live URL answers the question, and you update that URL in place. Keep the useful heading, revise the answer and note what changed. A near-duplicate starts its own discovery path while the page bots already fetch sits stale. If item 13 found competing posts, consolidate them first; this check is what keeps the same problem from returning on your next update.

## Before you buy another visibility dashboard

**Leave polish until access, discovery, rendering and extraction pass.** That means holding off on extra Organization markup, FAQ schema tuning, readability scores and authority scores while Tiers 1 to 4 still fail. None of those extras opens a blocked route or fills an empty HTML shell. A dashboard that monitors AI Overviews can show which prompt you lost. It cannot, by itself, repair the reason the bot never fetched the answer page.

I grade a tool on who does the fix. groas runs [technical fixes alongside new pages inside its SEO and AI Search Sprint](https://groas.com/for-businesses), with actions logged, rather than handing you another list of problems to manage. For this checklist, buy execution first and monitoring second.

**The item most often skipped is 4: check the edge before you blame the file.** Verify what bot traffic receives, not just what your own browser sees. Skip that check and the cost is ugly: content, schema and dates can all be ready while Cloudflare or AWS WAF keeps the bot out for months. I check Tier 1 before I touch a heading. You should too.

## Frequently Asked Questions

### Why does an AI visibility checklist start with robots.txt instead of schema?

Bots request access before they can judge a page, so a robots.txt mistake blocks everything downstream: no fetch means no answer to lift, entity to resolve or update date to notice. Schema gaps matter less because a bot that cannot reach the page never reads the markup at all.

### Do I have to allow GPTBot to show up in ChatGPT search results?

No. OpenAI assigns its crawlers different jobs: OAI-SearchBot serves search, while GPTBot is for training. You can allow search without allowing training, but check the actual rule group rather than assuming one robots.txt decision covers both. Allow roughly 24 hours for changes to propagate.

### How do I block Perplexity-User if it ignores robots.txt?

Perplexity-User generally ignores robots.txt because a person requested the page, so a robots.txt rule may not apply. Control those user-triggered fetches at the server or WAF instead, and check Perplexity's published IP guidance when configuring access. PerplexityBot, the indexing bot, is a separate concern.

### My robots.txt allows AI bots, so why are they still getting blocked?

Your firewall or edge may be the problem, not the file. Verify that permitted bot traffic gets a 200 response, not a 403 or a challenge. Check Cloudflare's Security > Bots and AI Crawl Control, or AWS WAF's managed bot rules and IP sets. Cloudflare announced AI-crawler blocking changes on July 1, 2025, and Perplexity publishes WAF whitelisting guidance.

### Do AI crawlers render JavaScript when reading my pages?

Not reliably. A crawler analysis discussed in the article found a JavaScript-rendering gap among major AI crawlers, so a client-side React shell that delivers text only after JavaScript runs is a bad bet. Test View Source, not just Inspect Element, and server-render article body, pricing facts and comparison tables.

### What should I do if several of my blog posts answer the same question?

Merge the overlapping posts and redirect the ones you retire with a 301, keeping one live URL for that question. Duplicates split the attention of a bot trying to choose what to fetch and cite. A site: search for the question should show one obvious answer page, not three near-duplicates.

### How much does freshness matter for AI citations?

Freshness matters only after access and rendering, since those decide whether the page can be read at all. In the Ahrefs 17M-citation analysis, AI Overviews cited pages about 16 days older on average than organic results, while ChatGPT skewed fresher. Keep the visible date, dateModified and sitemap lastmod consistent, and do not treat a fresh timestamp as a substitute for a reachable answer.

### Should I fix schema before making sure AI bots can actually fetch my pages?

No. Leave polish like extra Organization markup, FAQ schema tuning, readability scores and authority scores until access, discovery, rendering and extraction all pass. None of those extras opens a blocked route or fills an empty HTML shell. A monitoring dashboard can show which prompt you lost, but it cannot repair the reason the bot never fetched the page.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/the-ai-visibility-technical-checklist-21#webpage","url":"https://groas.com/post/the-ai-visibility-technical-checklist-21","name":"The AI Visibility Checklist: 21 Fixes in the Order Bots Fetch Your Site","description":"Run 21 technical SEO checks in fetch order, from robots.txt and edge access to answer pages and freshness. Stop at the first failure before polishing schema.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2f6bd5c0-8971-4b3e-b629-6b39020c2b09.png"},"breadcrumb":{"@id":"https://groas.com/post/the-ai-visibility-technical-checklist-21#breadcrumb"},"dateModified":"2026-10-16T05:38:48.664Z","mainEntity":{"@id":"https://groas.com/post/the-ai-visibility-technical-checklist-21#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/the-ai-visibility-technical-checklist-21#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"The AI Visibility Checklist: 21 Fixes in the Order Bots Fetch Your Site","item":"https://groas.com/post/the-ai-visibility-technical-checklist-21"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/the-ai-visibility-technical-checklist-21#article","url":"https://groas.com/post/the-ai-visibility-technical-checklist-21","mainEntityOfPage":{"@id":"https://groas.com/post/the-ai-visibility-technical-checklist-21#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"The AI Visibility Checklist: 21 Fixes in the Order Bots Fetch Your Site","description":"Run 21 technical SEO checks in fetch order, from robots.txt and edge access to answer pages and freshness. Stop at the first failure before polishing schema.","datePublished":"2026-10-16T05:38:48.601Z","dateModified":"2026-10-16T05:38:48.664Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/2f6bd5c0-8971-4b3e-b629-6b39020c2b09.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"pass fail, page, pass, fail, answer, bot, fetch, robots txt","wordCount":1926,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/the-ai-visibility-technical-checklist-21#faq","mainEntity":[{"@type":"Question","name":"Why does an AI visibility checklist start with robots.txt instead of schema?","acceptedAnswer":{"@type":"Answer","text":"Bots request access before they can judge a page, so a robots.txt mistake blocks everything downstream: no fetch means no answer to lift, entity to resolve or update date to notice. Schema gaps matter less because a bot that cannot reach the page never reads the markup at all."}},{"@type":"Question","name":"Do I have to allow GPTBot to show up in ChatGPT search results?","acceptedAnswer":{"@type":"Answer","text":"No. OpenAI assigns its crawlers different jobs: OAI-SearchBot serves search, while GPTBot is for training. You can allow search without allowing training, but check the actual rule group rather than assuming one robots.txt decision covers both. Allow roughly 24 hours for changes to propagate."}},{"@type":"Question","name":"How do I block Perplexity-User if it ignores robots.txt?","acceptedAnswer":{"@type":"Answer","text":"Perplexity-User generally ignores robots.txt because a person requested the page, so a robots.txt rule may not apply. Control those user-triggered fetches at the server or WAF instead, and check Perplexity's published IP guidance when configuring access. PerplexityBot, the indexing bot, is a separate concern."}},{"@type":"Question","name":"My robots.txt allows AI bots, so why are they still getting blocked?","acceptedAnswer":{"@type":"Answer","text":"Your firewall or edge may be the problem, not the file. Verify that permitted bot traffic gets a 200 response, not a 403 or a challenge. Check Cloudflare's Security > Bots and AI Crawl Control, or AWS WAF's managed bot rules and IP sets. Cloudflare announced AI-crawler blocking changes on July 1, 2025, and Perplexity publishes WAF whitelisting guidance."}},{"@type":"Question","name":"Do AI crawlers render JavaScript when reading my pages?","acceptedAnswer":{"@type":"Answer","text":"Not reliably. A crawler analysis discussed in the article found a JavaScript-rendering gap among major AI crawlers, so a client-side React shell that delivers text only after JavaScript runs is a bad bet. Test View Source, not just Inspect Element, and server-render article body, pricing facts and comparison tables."}},{"@type":"Question","name":"What should I do if several of my blog posts answer the same question?","acceptedAnswer":{"@type":"Answer","text":"Merge the overlapping posts and redirect the ones you retire with a 301, keeping one live URL for that question. Duplicates split the attention of a bot trying to choose what to fetch and cite. A site: search for the question should show one obvious answer page, not three near-duplicates."}},{"@type":"Question","name":"How much does freshness matter for AI citations?","acceptedAnswer":{"@type":"Answer","text":"Freshness matters only after access and rendering, since those decide whether the page can be read at all. In the Ahrefs 17M-citation analysis, AI Overviews cited pages about 16 days older on average than organic results, while ChatGPT skewed fresher. Keep the visible date, dateModified and sitemap lastmod consistent, and do not treat a fresh timestamp as a substitute for a reachable answer."}},{"@type":"Question","name":"Should I fix schema before making sure AI bots can actually fetch my pages?","acceptedAnswer":{"@type":"Answer","text":"No. Leave polish like extra Organization markup, FAQ schema tuning, readability scores and authority scores until access, discovery, rendering and extraction all pass. None of those extras opens a blocked route or fills an empty HTML shell. A monitoring dashboard can show which prompt you lost, but it cannot repair the reason the bot never fetched the page."}}]}]}
```