On a Wednesday in June 2026, without an announcement, a press conference or a single line in your analytics, your website's audience changed species. Cloudflare's measurement network — which sits in front of roughly a fifth of the web — reported that 57.3% of the world's HTTP requests for HTML content were coming from machines, and 42.7% from people. Cloudflare's own CEO had told an audience in March that this crossover would happen in 2027. It arrived early.
Most of those machines don't matter to you. Two-thirds of AI traffic is training crawlers hoovering up text for the next model — background weather. But a small, violently fast-growing slice is something genuinely new: an AI agent standing at your door, right now, on behalf of a specific person who is trying to get something done. That agent has a budget, a deadline and no patience. And it doesn't see your website the way you do.
This piece is about that visitor. Everything in it is checkable, every number links to its source, and the practical part at the end costs nothing and takes twenty minutes. Nothing to install, nothing to buy.
THE SHORT VERSION, IF YOU ONLY READ ONE BOX
- Bots passed humans in June 2026: 57.3% of requests for HTML, versus 42.7% from people.
- "Bot" is now three different animals. Cloudflare splits them into Search, Agent and Training — and only the Agent has a human waiting for the answer.
- Agent traffic grew 7,851% year over year. Most of it now arrives inside AI browsers, not crawlers, so
robots.txtis the wrong lever. - Agents read your page as an accessibility tree, not a picture. A button with no accessible name does not exist for them.
- They fail a lot — the best agent tested on live sites finishes 61.3% of tasks — and when they fail on you, nothing tells you. They just leave.
- The fix is not a new product category. It is old, boring, human-friendly web hygiene, and it is free.
1. The number that changed the audience
Here is the crossover, drawn to scale. It is the first time in the history of the web that this bar has looked like this.
Resist the first instinct. The instinct says: machines are eating my website, put up a wall. A great many businesses acted on that instinct somewhere between 2023 and 2025, usually by flipping one switch in a dashboard, and most of them have no idea what that switch actually did. Because by 2026 the word "bot" had stopped meaning one thing.
2. Three machines knock. Only one of them is a customer.
On 1 July 2026 Cloudflare quietly did the most useful thing anyone has done for site owners in years: it stopped treating AI traffic as a single category and split it into three behaviours you can control separately. Their definitions, verbatim:
The librarian
"Crawlers that index your content so they can answer questions about it later, where you should expect referral traffic or other equitable compensation in return."
IF YOU SHUT THIS DOOR:You are not in the index. You cannot be retrieved, so you cannot be recommended, so the question never reaches you at all.
The customer's proxy
"Automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents."
IF YOU SHUT THIS DOOR:Someone already asked about you and their agent got a closed door. This is the only one of the three with a human waiting on the other end.
The archivist
"Crawlers that take your content to train or fine-tune a model."
IF YOU SHUT THIS DOOR:A defensible choice with real arguments on both sides. It costs you close to nothing today, and it is two-thirds of all AI-driven traffic by volume.
Read the middle column again. That is not a scraper. That is a person — a facilities manager pricing a fit-out, a parent finding a paediatric dentist, an ops lead shortlisting three studios — who has delegated the looking. The agent is the customer's hands.
The reason it goes undiscovered is structural. A blocked agent produces no bounce, no form abandonment, no angry email. Your analytics record nothing, because nothing loaded. The failure is perfectly silent, which is the worst property a failure can have.
3. The category that grew 7,851%
HUMAN Security analysed more than a quadrillion digital interactions for its 2026 benchmark on AI traffic, published in April. The headline everyone repeated was that automated traffic is growing about eight times faster than human traffic — automation up 23.51% year over year against 3.10% for people. That is not the interesting number.
This is the interesting number:
BARS ARE ON A COMPRESSED SCALE — A TRUE-TO-SCALE 7,851% BAR WOULD BE 330 METRES LONG AT THIS WIDTH. THE POINT IS THE ORDER OF MAGNITUDE, NOT THE PIXELS.
Now the detail that breaks most published advice. That agent traffic is not arriving as a polite crawler with a documented user-agent. It is arriving inside a browser. In May 2026, agentic browsers made up the overwhelming majority of agent activity:
An agentic browser is a real browser. It runs JavaScript, holds cookies, carries the user's session, and looks — to most of your tooling — like a slightly odd Chrome. It is not fetching your robots.txt and asking permission, because it is not a crawler; it is a person's browser with a machine at the wheel. Every checklist that ends at "add these user-agents to robots.txt" stops one step short of the visitor that matters.
4. What it actually does once it's inside
Here is where the hype and the data part company, and this article is going to side with the data even though it makes for a smaller story. Agents are not swarming your checkout. In May 2026, three quarters of everything agents did on the web was looking:
So: 2.4% is a rounding error, and anyone telling you that agents are about to become your main sales channel is selling something. But look at what 76.4% means. The agent is doing the shortlisting. It is the research phase — the exact phase where a business gets included or quietly dropped, three days before anyone talks to a human. You are not being bought from by a robot. You are being screened by one.
And screening is where you can still lose without ever knowing you were in the running.
5. They fail more often than the demos suggest
Every launch video shows an agent booking a flight in one clean pass. Then researchers put agents on real, live, unmodified websites, and the numbers collapse. Online-Mind2Web is the benchmark that did this honestly: 300 tasks across 136 live sites, graded by humans.
Now put those two facts next to each other. Roughly two in five agent attempts fail on live websites, and roughly one in eleven agent requests is being blocked outright before it starts. Both of those failures are silent on your end.
A human who can't find your price gets annoyed, maybe calls. An agent that can't find your price returns to its user with two competitors and a sentence about how it couldn't confirm yours. You are not rejected. You are omitted, which is worse, because there is nothing to fix if you never learn it happened.
6. The part almost nobody tells you: it isn't looking at your page
This is the section to read twice, because it quietly rewrites what "a good website" means.
You probably assume an AI browsing agent takes a screenshot and looks at it, the way you would. Some do, some of the time. But the dominant, cheaper, faster and far more reliable method is to skip the pixels entirely and read the accessibility tree — the structured description of a page that browsers generate for screen readers. Microsoft's own Playwright MCP server, one of the most widely used pieces of agent plumbing in the world, states it in its README without a hint of ambiguity: it "uses Playwright's accessibility tree, not pixel-based input", and needs "no vision models" because it "operates purely on structured data".
So here is your homepage, twice. Left: what your designer signed off. Right: what a large share of agents actually receive.
PRICE IS IN THE HERO IMAGE. PHONE NUMBER IS IN THE HEADER GRAPHIC. THE THREE SERVICE TILES ARE ICONS.
NO PRICE. NO PHONE. NO SERVICES. NO CLICKABLE CONTROL. FROM HERE THE AGENT CANNOT ANSWER A SINGLE QUESTION ABOUT THIS BUSINESS.
div that happens to have a click handler.Sit with the consequence, because it is genuinely good news. Everything that makes your site work for an agent is something that already makes it work for a blind customer. Real headings. Buttons that are buttons. Form fields with labels attached. Alt text that says what the picture says. Keyboard reachability. There is no new discipline to buy here — the discipline is thirty years old, it has a standard, and the industry has been half-ignoring it for a decade.
There's a second class of agent that never even gets a browser: the plain fetch tool inside an assistant, which pulls raw HTML and reads it as text. Anthropic documents that its web fetch tool does not support JavaScript-rendered pages at all. For that visitor, whatever your framework paints after load simply isn't there. Two very different machines, one shared requirement: put the facts in the markup.
A SMALL THING THAT HAPPENED WHILE WRITING THIS
Researching this article, my fetching tool asked HUMAN Security's own page about agentic traffic for its contents and got 403 Forbidden. A plain command-line request to the identical URL, in the same minute, got 200 OK. Nobody did anything wrong; that is just a bot policy doing its job, and their figures were readable elsewhere. But it is a perfect miniature of the whole problem: the door is answered differently depending on who knocks, the knocker gets no explanation, and the site owner never finds out.
7. The door test — seven checks, twenty minutes, no tools
None of this needs a vendor. Do it yourself this afternoon, on the three pages that actually earn you money.
Open yoursite.com/robots.txt and actually read it. You're looking for named AI user-agents sitting under a blanket Disallow: /.
Your CDN or WAF outranks your robots file. On Cloudflare, look for the AI traffic controls and check Search, Agent and Training separately. On other platforms, look for anything named "block AI bots".
IF IT FAILS: one toggle flipped in 2024 is still quietly turning away every customer's agent in 2026.Disable JS in your browser settings and open your most important page. Whatever remains is what a fetch-only agent gets.
IF IT FAILS: your prices, services and phone number don't exist for an entire class of assistant. Server-render the facts, even if the rest stays dynamic.Hands off the mouse. Tab, Enter, Space. Get from your homepage to a submitted enquiry, booking or order.
IF IT FAILS: exactly where you get stuck is where the agent gets stuck — it is navigating the same tree your keyboard is.Private window, homepage, ten seconds. Count everything that appears uninvited: cookie wall, newsletter modal, chat bubble, region picker, discount popup.
IF IT FAILS: every layer is a guess the agent has to make before it reaches a single fact about you. Modals interrupting mid-flow are a documented top cause of agent failure.Drag your cursor across your price, your phone number, your opening hours, your service area. If the text doesn't highlight, it's a picture.
IF IT FAILS: it isn't data, it's decoration. Nothing that lives only inside an image, a PDF or a canvas element makes the trip.Buttons, links, form fields. "Book a call", "Email address", "Download the price list" are names. A bare icon, a div with a click handler, and a field labelled only by its placeholder are not.
One command, no install, macOS or Linux. It strips the tags and shows you the plain text an assistant receives. Read the first thirty lines out loud — if the answer to "what do you sell, where, for how much" isn't in there, that's your afternoon's work.
If that command prints your headline, your services, a price and a phone number, you are already ahead of most of your competitors. If it prints "Loading…" or nothing at all, you have found the single most valuable thing on this page.
And the file that fixes half of it
If your robots.txt is a mess, this is a safe, explicit starting point. It lets in the machines that bring you customers, and leaves the training decision to you as an actual decision:
One trap worth knowing, because half-written robots files do more damage than missing ones: a crawler obeys only the group named after it and ignores User-agent: * entirely once such a group exists. Every rule you cared about has to be repeated inside the named group.
| User-agent | Operator | Behaviour | What it costs you to block it |
|---|---|---|---|
| ChatGPT-User | OpenAI | Agent · live | A person asked ChatGPT about you and it couldn't open your page. |
| Claude-User | Anthropic | Agent · live | Same, in Claude. This is the one that costs actual jobs. |
| Perplexity-User | Perplexity | Agent · live | Same, in Perplexity. |
| OAI-SearchBot | OpenAI | Search · index | You're not retrievable in ChatGPT search, so you can't be recommended. |
| Claude-SearchBot | Anthropic | Search · index | Same, in Claude. |
| PerplexityBot | Perplexity | Search · index | Same, in Perplexity. |
| Googlebot | Search · index | You leave Google Search. There is no version of this that is a good idea. | |
| Google-Extended | Training | Opts you out of Gemini training and grounding. Does not remove you from AI Overviews. | |
| GPTBot | OpenAI | Training | Your text stays out of future training data. Defensible. Costs little today. |
| ClaudeBot | Anthropic | Training | Same decision, same trade. |
8. What not to spend money on
⚠ FOUR THINGS BEING SOLD RIGHT NOW THAT DON'T WORK
"Agent-readiness certification." There is no registry, no submission form and no certificate that any assistant checks. The agent visits your live site and finds out for itself.
llms.txt. Eighteen months into the conversation, no major model vendor documents consuming an external llms.txt from your domain, and Google has publicly said it does not support it. We covered why it turned out to be a dead end. It costs ten minutes, so add it if you like — just don't count it as work done.
"GEO instead of SEO." Retrieval runs on a search index. Being findable by a search engine is not the old game you've graduated from; it is the entry ticket to the new one.
Blanket "block all AI" toggles. They were built when there was one kind of machine. There are three now, and the toggle can't tell your customer's proxy from a scraper.
9. The honest caveat, and why this still matters if you don't sell online
Let me argue against my own headline for a paragraph, because the strongest version of this case has to survive the objection.
Agentic traffic fell 4.3% month over month in May. Only 2.4% of it touches checkout. More than 95% of AI-driven automation is concentrated in three verticals — retail, media, travel — and if you run a specialist consultancy in Utrecht, you are not in any of them yet. The commerce protocols making the headlines, OpenAI's Agentic Commerce Protocol (September 2025, built with Stripe) and Google's Universal Commerce Protocol (announced at NRF on 11 January 2026, with Shopify, Etsy, Target and Walmart), are merchant infrastructure. If you don't have a cart, they are not your problem this year.
All true. And none of it changes what you should do on Monday, for one reason: every single item in the door test is something a human customer benefits from too. A page that states its price in text is better for the person on a train with bad signal. A form you can complete with a keyboard is better for the customer with a tremor. A homepage without four stacked popups is better for everyone who has ever visited a homepage.
This is the rarest shape a technology trend can take: the speculative future and the boring present ask for exactly the same work. You are not making a bet on agents. You are fixing your website, and getting agent-readiness for free.
The short version
The majority of requests hitting the web are now machines, and most of them don't matter. One kind does: the agent acting live for a person who is trying to choose someone. It arrives inside a browser, reads your page as a structured tree rather than a picture, fails about two times in five, and never tells you when it failed.
You can't buy your way onto its shortlist. You can only be legible to it — and being legible to a machine turns out to be identical to being legible to a person in a hurry, in the dark, on a bad connection, with a screen reader. Twenty minutes, seven checks, no invoice.
Most of your competitors will read a headline about robots taking over the internet, feel vaguely alarmed, and change nothing. That is the whole opportunity, and like all such opportunities it has a shelf life.
HOW THIS ARTICLE WAS MADE
Every figure above is linked to its source below, and every source is primary or one hop from it: Cloudflare's own changelog for the three-category split, HUMAN Security's published benchmark for the traffic figures, the Online-Mind2Web paper for the success rates, and Microsoft's Playwright MCP README for the accessibility-tree quote. Where a number is soft, jumpy or one month old, this piece says so rather than rounding it into confidence. Landingpeak is not affiliated with Cloudflare, HUMAN Security, OpenAI, Anthropic, Google, Perplexity or Microsoft.