A Citation Starts With a Fetch: Why AI Answers Miss Your Page

A citation starts with a page the engine can read

You cannot get a citation from a page the answer engine cannot read. Most agencies won’t lead with that because it makes content packages harder to sell. But when a business is missing from an AI answer, I would check the retrieval path before paying someone to rewrite the page.

I learned that habit debugging Quality Score. When a keyword sat at 3/10, I stopped rewriting ads first and checked whether the landing page loaded, matched the query and gave Google text it could understand. The analogy has a limit: Quality Score and AI citations are different systems. The diagnostic habit carries over. Find where the signal dies before you blame the copy.

Say you run a garage door repair company in Dallas, spend $20k a month on search, and a buyer asks ChatGPT who fixes a broken garage door spring in Dallas without a same-day surcharge. For your page to supply a cited answer, the retrieval system has to know the URL, obtain the page, extract the relevant text, find the passage that answers the question and attribute that answer to you. A failure at any earlier stage keeps your prose out of the later contest.

We will use that garage door page to follow the chain. The order matters: an excellent paragraph cannot rescue a URL the engine never reaches.

First, separate remembered knowledge from retrieved pages

An AI system can know about your business in two different ways. The first is training data: text the model encountered before the buyer asked anything. That knowledge is slow to change and does not give you a reliable route to a citation next week. You cannot edit your Dallas page today and assume the model’s memory will update tomorrow.

The second is retrieval. For an answer that cites current pages, the system looks for sources it can use in response to the question. That is the path a page owner can inspect. I used to tell clients that getting content into the model’s long-term memory was the prize. I was wrong about how useful that framing was for a business trying to answer a buyer’s specific question now.

OpenAI separates these jobs across GPTBot for training, OAI-SearchBot for search indexing and ChatGPT-User for user-initiated page visits. Blocking GPTBot does not, by itself, remove you from ChatGPT answers, while opting out of OAI-SearchBot prevents a site from appearing in ChatGPT search answers. Perplexity likewise distinguishes PerplexityBot, which helps surface linked results, from Perplexity-User, which visits pages for a user’s question.

That distinction gives us the first useful question: which route to the answer is broken? Checking a training crawler when the problem is search visibility will waste your afternoon. I have wasted afternoons on less.

Stage 1: the engine must discover the URL

Before the Dallas spring-repair page can become a source, the retrieval system needs a way to find it. This is old plumbing: a crawlable link path, a useful sitemap and internal links pointing to the page. Google’s AI Overviews require a page to be indexed and eligible to show a snippet; there is no separate AI index to submit it to. Retrieval can also use query fan-out, searching multiple versions of the buyer’s question.

Our buyer did not type the title of your service page. They asked about a spring repair, Dallas and a surcharge. If your location page has no incoming internal links or is missing from the index, that page has little chance of entering the candidate set, however neatly it answers the question. For ChatGPT and Perplexity, their search bots do discovery work ahead of a user-initiated visit. The visit cannot help much if the URL never becomes a candidate.

Do not mistake this for a demand to publish more pages. A new article about garage door springs does not fix an undiscoverable service page. Check whether the existing URL can be found before you rewrite its headline.

Stage 2: a discovered page still has to survive the fetch

Now the engine has a candidate URL. It still needs the page. This is where an ordinary infrastructure decision can erase good content: a robots.txt disallow, a CDN rule aimed at AI agents, a firewall challenge or a slow page that fails before its useful content arrives. From the buyer’s side, there is no dramatic error screen. The answer may simply use another source.

These controls do not affect every bot identically. ChatGPT-User visits are user-initiated, so robots.txt rules may not apply to them in the same way as automatic crawling. Perplexity-User generally ignores robots.txt. Indexing bots matter separately: block OAI-SearchBot or PerplexityBot and you undermine the route by which your URL joins their search candidates. After changing robots.txt, allow about 24 hours for OpenAI’s systems to adjust; Perplexity describes a similar delay.

The edge deserves its own look. Cloudflare’s AI-bot controls distinguish Search, Agent and Training traffic, so a rule described casually as an “AI block” may have consequences its owner did not intend. Broken URLs matter too: one study found 34% of AI fetches went to 404s, versus 8% for Googlebot. That is not a reason to chase every bot request. It is a reason to check what happened to the URL that should answer this buyer.

Read verified requests, not costumes

A user-agent string is not proof that an AI service visited your page. Anyone can label a request GPTBot or ChatGPT-User, and scrapers do. Verification requires more than matching that string; IP-range checks matter. OpenAI publishes per-bot JSON files, and Google’s verification uses multiple signals, including reverse DNS. The edge-log evidence on verified and spoofed traffic is a useful antidote to reports that count every self-declared bot hit as a citation opportunity.

For the Dallas page, I would inspect verified requests, their status codes and what the server returned. A verified request that receives a 403 tells a different story from one that receives the page. No verified request does not, on its own, prove that a firewall blocked it: the URL may not have been selected for a visit at all. Use the logs to locate a failure, not to invent one.

Stage 3: fetching HTML is not the same as reading the offer

Suppose the request succeeds. The engine receives your page, but the text it can extract may not match what you see in a browser. Major AI crawlers may download JavaScript without executing it. If essential pricing or product details appear only after client-side rendering, a crawler reading the initial response can miss them.

On our Dallas page, imagine the flat-rate spring price lives in a quote widget, the same-day condition appears after a click and the review count loads separately. To a buyer browsing the page, that looks like one coherent offer. In the fetched HTML, it may be an empty container and a few labels. The parser cannot cite a price it never receives.

Start with the raw page response. Search for the words the buyer needs: garage door spring, Dallas, same-day surcharge and the condition attached to the offer. Then check whether navigation, overlays or repeated boilerplate bury those words. Tabs and accordions are not automatically fatal; the question is whether their answer text is present and extractable in the fetched HTML.

If the offer is absent from the response, fix rendering before copy. That is why my pre-content check covers crawler rules, edge logs and rendering before anyone briefs another article.

Stage 4: the answer must fit in a usable passage

Once a system extracts text, it has to find the part relevant to the question. Retrieval commonly works with chunks rather than asking the model to treat every page as one indivisible document. If no retrieved chunk contains the answer, that answer cannot be used as support. Teams may work with roughly 400–800-token chunks and some overlap, but the exact size is not the writing target. Keeping the necessary facts together is.

The buyer needs to know four things: the service, the city, whether there is a same-day surcharge and any condition on that promise. Put one fact in a hero line, another in a closed FAQ tab and the condition in the footer, and the page may produce three passages that each answer only part of the question.

I would put the answer under a heading that names the question, then make the first sentence stand on its own. For example: “We do not charge a same-day surcharge for garage door spring repair in Dallas on standard torsion springs booked before 2pm.” Follow it with the flat rate and its boundary, if those are the actual terms. Do not borrow the sentence unless it is true for your business. The point is the structure: a reader and a retriever should not have to assemble your offer from three corners of the page.

The same applies to a price table. Separating a table row from its footnote can destroy the meaning. Keep the price and the condition close enough to travel together. Stop grading only the page; grade the passage that could answer the question.

Stage 5: a usable passage still has to earn attribution

A page that survives discovery, fetching, parsing and passage selection has a chance at a citation, not a guarantee. Other passages may answer the same question more directly. Relevance, entity alignment, freshness and originality can influence which source gets used. The practical question is whether a passage supplies a clear, attributable answer rather than a mood.

Compare two illustrative Dallas pages. One says “affordable same-day service.” Another says it repairs torsion springs in Dallas for a $189 flat rate, with no same-day surcharge on jobs booked before 2pm, and shows when that information was checked. The second gives the system a specific claim to attribute. The first gives it an adjective. I am not telling you to invent a price, a deadline or a revision date to win a link. I am telling you to state the real terms where they can be read together.

Freshness helps only when it reflects something useful. An updated date cannot rescue a vague offer, and a precise offer cannot rescue a blocked fetch. Attribution is the last stage to improve, not the first excuse for buying more content.

Find the broken stage before choosing the fix

Most citation advice starts with clear answers, schema and authority. Those may help, but they are poor first moves when the page is blocked, broken or missing its answer in raw HTML. I check in pipeline order because each stage gates the next:

The table is a troubleshooting order, not a promise that every uncited page has exactly one fault. Your Dallas page could be both blocked at the edge and missing its price in the HTML. Fixing the block reveals the next problem. That is progress, even if the citation does not appear immediately.

Run the model on a page we have not discussed

Say you sell invoicing software and Perplexity never cites your $29 plan page. Do not rewrite the headline yet. First, check that the page can enter discovery. Then look for verified PerplexityBot requests and inspect any response you find. If the fetch succeeds, search the returned HTML for “$29” and the plan terms.

Suppose the price is absent because a pricing toggle inserts it after the page loads. That is a parsing problem. Suppose the price is present, but the seat limit sits in another section and the annual-billing condition appears in a footnote. Now inspect whether one usable passage contains all three. If it does, you can finally compare the clarity and specificity of that passage with the sources being cited. You did not need a special theory of invoicing software to reach any of those decisions. You followed the chain.

That is why I stopped treating AI citations as a content project alone. The live-fetch checklist turns the sequence into a routine; groas runs this kind of search work continuously instead of leaving it to a quarterly blog rewrite. Start with the earliest broken stage. The copy can wait.