How Claude reads your website

Someone opens Claude and types "who should we hire to migrate our billing system?" Or "best pediatric dentist in Portland." Or "which studio actually knows Webflow." What happens in the next four seconds is not a mystery, not a black box, and not something an agency needs to reverse-engineer for you. Anthropic publishes it. The crawler names, the retrieval order, the length of the snippet it quotes, the file types it can and cannot read — it's all in the developer documentation, in public, for free.

Almost nobody in the GEO industry reads it. So this piece does the boring thing: it takes Anthropic's own docs and turns them into things you can change on your website this week.

1. Claude sends three different robots. Most sites block the wrong one.

This is the single most expensive mistake I find on websites of every size, and it takes ninety seconds to fix. Anthropic runs three separate crawlers, each with its own user-agent, each blockable independently in robots.txt. They do completely different jobs.

Now the trap. Sometime in the last two years, someone turned on the "block AI scrapers" switch in Cloudflare, or installed a WordPress plugin promising to "protect your content from AI." Those switches are blunt. They block all three. The intent was to stop training scrapers; the effect is that when a prospect asks Claude about you specifically, Claude knocks and gets a closed door.

Blocking ClaudeBot is a real decision with real arguments on both sides. Blocking Claude-User is not a decision anyone makes on purpose. Go and look at yours right now:

One more thing worth knowing, because it trips people up: a crawler obeys only its own named group and ignores User-agent: * entirely once a group with its name exists. So if you add a block for one bot, every rule you cared about has to be repeated inside that group. Half-written robots.txt files do more damage than missing ones.

2. Claude cannot type your URL from memory

Here's a constraint with enormous consequences that almost nobody talks about. For security reasons, Anthropic's fetch tool can only retrieve URLs that have already appeared in the conversation — pasted by the user, or returned by a previous web search. Claude is not allowed to invent a URL and go there.

Read that again, because it dismantles a whole category of GEO advice. You cannot "submit" yourself to Claude. You cannot put a magic file at the root of your domain and wait to be found. If Claude is going to read your page, one of two things has to happen first: a human pastes your link, or a search brings you back.

Step two is why "GEO instead of SEO" is a false choice sold by people with a new invoice line to justify. Retrieval runs on a search index. Being findable by a search engine is not the old game you've moved on from — it is the entry fee for the new one.

3. The fetch tool cannot read a JavaScript website. At all.

Anthropic's documentation says it in one flat sentence: the web fetch tool does not support websites dynamically rendered with JavaScript. Supported content types are text, HTML and PDF — anything else comes back as an error.

Translate that into a real website. If your pricing table is drawn by a React component after the page loads, Claude receives an empty shell. If your product list lives inside a page-builder widget that renders client-side, it isn't there. If your headline claim or your phone number is baked into an image because the designer liked the font, it does not exist. Not "ranks lower" — does not exist.

The test costs nothing. Run it, then search the output for your own price, your product name, your phone number:

An empty result is the most important negative finding in your whole marketing stack. And it explains something that confuses a lot of owners: why a beautiful, expensive, award-winning site gets ignored while a competitor's ugly 2016 HTML page keeps getting named. The ugly one is legible.

4. One hundred and fifty characters. That's the unit.

Every citation the Claude API returns from a web search includes a field called cited_text — up to 150 characters of the source page. That snippet is the piece of your writing that actually travels: it's what the model anchors its claim to, and in a lot of interfaces it's what the human sees on hover.

So stop writing paragraphs and start writing quotable units. Watch what happens to the same business described twice.

Four rules follow directly from that 150-character window, and they're the most useful thing in this article:

1. One sentence, one claim

   A sentence carrying three ideas can't be lifted cleanly. Split it. Each sentence should survive being torn out of the page and read alone by a stranger.

2. Put a number in it

   A price, a range, a year, a response time, a licence number, a review count. Numbers are what make a sentence worth quoting instead of paraphrasing — and a paraphrase doesn't come with a link to you.

3. No orphan pronouns

   "We also do that for most of the other platforms" is dead weight — the snippet travels alone, and "that" doesn't come with it. Name the service and the platform again. It reads slightly repetitive to a human and perfectly to a machine.

4. Answer in the first 200 words

   Filtering happens before Claude reads. A page that opens with a founder's story and hides the price under three testimonials gets discarded before anything you wrote is considered.

5. Freshness isn't a vibe. It's a field.

Here's what a single search result looks like when it arrives — this is the actual shape of the object, from Anthropic's docs:

Four fields. One of them is your URL, one is your title tag, one is the page itself — and one is a date. Page age is not something the model infers from your tone; it's structured data sitting right next to your headline. When two competitors say roughly the same thing and one page was touched this spring while the other hasn't moved since 2021, you don't need a theory about what happens next.

This is also the cheapest lever on the whole list. Put a real, visible "Updated August 2026" on your service and pricing pages, make your dateModified honest, and revisit prices quarterly. Not a rewrite — a revision. Changing a price range from last spring's numbers to this month's numbers is a better use of an hour than any blog post you'll ever publish.

6. Claude is built to under-claim. That's your opening.

In January 2026 Anthropic published Claude's Constitution — the document describing the values the model is trained toward. Two lines from it matter more to your marketing than anything in a GEO deck: Claude aims to have calibrated uncertainty based on evidence and sound reasoning, and to avoid conveying beliefs with more or less confidence than it actually has.

Sit with what that does to a homepage. "The best digital agency in New York" is a claim the model has no evidence for, from a source with an obvious commercial interest, on a page that also wants to sell something. Repeating it would mean asserting something it can't support. So it doesn't. Your superlatives aren't neutral filler — they're the part that gets left behind.

Meanwhile: "Founded 2016. 412 reviews at 4.8 average as of August 2026. SOC 2 Type II since March 2024." That's checkable. A model can repeat it without exceeding what it knows. It reads as less confident and gets quoted more.

7. Half of it isn't on your website at all

Everything above is about the page. But retrieval pulls from an index of the whole web, and the strongest published signal is what other people's pages say about you. Ahrefs ran the largest study on this to date — 75,000 brands, correlated against AI visibility:

Correlation isn't causation, and the study's authors say so. But the mechanism is easy to believe once you've seen the pipeline: a link is a vote a machine has to interpret; a sentence in somebody else's article — "Northwind ran the data migration for the county library system" — is an answer a machine can lift whole, from a source with no reason to flatter you.

If your business name appears on exactly one domain in the world, there is exactly one document to retrieve, and it's the one selling something. Partner and integration pages, industry association directories, the trade press, a conference speaker bio, a subreddit answer, a captioned YouTube walkthrough — those are the documents that corroborate you. Boring, slow, unglamorous, and worth more than another 1,500-word blog post on your own site.

8. The traffic will look tiny. Ignore that.

Anyone who does this work runs into the same objection: the analytics number is embarrassing. Ahrefs published their own: AI search sent them 0.5% of visitors — and 12.1% of signups. Roughly twenty-three times the conversion rate of everything else.

That ratio makes sense the moment you picture the person. They didn't scroll ten blue links. They asked a question, got three names with a reason attached to each, and clicked one. They arrive already recommended. If a single new customer is worth four figures to you, forty of those visitors a month is not a rounding error — it's the month.

The nine things, in order

• Open yoursite.com/robots.txt and confirm Claude-User and Claude-SearchBot aren't blocked
• curl your homepage and confirm your prices and phone number are in the raw HTML
• Get the phone number out of the image and into text
• Rewrite your top three pages so the first 200 words answer the question in the title
• Rewrite five sentences to be under 150 characters, each with one fact and one number
• Publish a real price range — a range with two numbers in it, not "call for pricing"
• Put a visible "Updated [month] 2026" on service and pricing pages, and keep it true
• Get named in three places you don't own this quarter — a partner, an association, the trade press
• Re-run the check in six weeks, because your competitors' pages move too

The short version

Claude isn't grading your website. It's looking for a sentence it can repeat without being wrong, on a page it's allowed to open, in a format it can parse, from a business other people also talk about. Every single one of those four conditions is documented, and every single one is something you can fix without permission from anyone.

Most of your competitors will never read the docs. That's the whole opportunity, and it has a shelf life.