On a Wednesday in June 2026, without an announcement, a press conference or a single line in your analytics, your website's audience changed species. Cloudflare's measurement network — which sits in front of roughly a fifth of the web — reported that 57.3% of the world's HTTP requests for HTML content were coming from machines, and 42.7% from people. Cloudflare's own CEO had told an audience in March that this crossover would happen in 2027. It arrived early.

Most of those machines don't matter to you. Two-thirds of AI traffic is training crawlers hoovering up text for the next model — background weather. But a small, violently fast-growing slice is something genuinely new: an AI agent standing at your door, right now, on behalf of a specific person who is trying to get something done. That agent has a budget, a deadline and no patience. And it doesn't see your website the way you do.

This piece is about that visitor. Everything in it is checkable, every number links to its source, and the practical part at the end costs nothing and takes twenty minutes. Nothing to install, nothing to buy.

THE SHORT VERSION, IF YOU ONLY READ ONE BOX

  • Bots passed humans in June 2026: 57.3% of requests for HTML, versus 42.7% from people.
  • "Bot" is now three different animals. Cloudflare splits them into Search, Agent and Training — and only the Agent has a human waiting for the answer.
  • Agent traffic grew 7,851% year over year. Most of it now arrives inside AI browsers, not crawlers, so robots.txt is the wrong lever.
  • Agents read your page as an accessibility tree, not a picture. A button with no accessible name does not exist for them.
  • They fail a lot — the best agent tested on live sites finishes 61.3% of tasks — and when they fail on you, nothing tells you. They just leave.
  • The fix is not a new product category. It is old, boring, human-friendly web hygiene, and it is free.

1. The number that changed the audience

Here is the crossover, drawn to scale. It is the first time in the history of the web that this bar has looked like this.

WHO IS ASKING FOR YOUR PAGES · JUNE 2026 SOURCE: CLOUDFLARE RADAR
HUMANS · 42.7% MACHINES · 57.3% 42.7 57.3 People, browsers, phones Crawlers, fetchers, agents MARCH 2026 — CLOUDFLARE'S CEO PREDICTS THE CROSSOVER FOR 2027 JUNE 2026 — IT HAPPENS
Share of worldwide HTTP requests for HTML content, reported from Cloudflare Radar in June 2026. Requests, not visitors: one agent can fire hundreds of requests where a person fires five. That is exactly why the raw share is the wrong thing to panic about — and the composition underneath it is the right thing to look at.

Resist the first instinct. The instinct says: machines are eating my website, put up a wall. A great many businesses acted on that instinct somewhere between 2023 and 2025, usually by flipping one switch in a dashboard, and most of them have no idea what that switch actually did. Because by 2026 the word "bot" had stopped meaning one thing.

2. Three machines knock. Only one of them is a customer.

On 1 July 2026 Cloudflare quietly did the most useful thing anyone has done for site owners in years: it stopped treating AI traffic as a single category and split it into three behaviours you can control separately. Their definitions, verbatim:

THE THREE MACHINES · AND WHAT SHUTTING EACH DOOR ACTUALLY COSTS YOU DEFINITIONS: CLOUDFLARE, 1 JULY 2026
AGENT

The customer's proxy

"Automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents."

IF YOU SHUT THIS DOOR:
Someone already asked about you and their agent got a closed door. This is the only one of the three with a human waiting on the other end.
TRAINING

The archivist

"Crawlers that take your content to train or fine-tune a model."

IF YOU SHUT THIS DOOR:
A defensible choice with real arguments on both sides. It costs you close to nothing today, and it is two-thirds of all AI-driven traffic by volume.
Cloudflare offers three settings per category: allow, block everywhere, or block only on pages that display ads. From 15 September 2026, newly onboarded domains get Training and Agent blocked on ad-bearing pages by default, with Search left allowed. Most small business sites carry no ads and are untouched — but the direction of travel is clear: your door is becoming a policy rather than an accident.

Read the middle column again. That is not a scraper. That is a person — a facilities manager pricing a fit-out, a parent finding a paediatric dentist, an ops lead shortlisting three studios — who has delegated the looking. The agent is the customer's hands.

Blocking the training crawler is a decision. Blocking the agent is a mistake nobody makes on purpose, and almost nobody discovers.

The reason it goes undiscovered is structural. A blocked agent produces no bounce, no form abandonment, no angry email. Your analytics record nothing, because nothing loaded. The failure is perfectly silent, which is the worst property a failure can have.

3. The category that grew 7,851%

HUMAN Security analysed more than a quadrillion digital interactions for its 2026 benchmark on AI traffic, published in April. The headline everyone repeated was that automated traffic is growing about eight times faster than human traffic — automation up 23.51% year over year against 3.10% for people. That is not the interesting number.

This is the interesting number:

YEAR-OVER-YEAR GROWTH · WHO IS ACTUALLY ARRIVING SOURCE: HUMAN SECURITY, 2026 STATE OF AI TRAFFIC
AI agents and agentic browsers — acting for a person, live +7,851%
AI scrapers — collecting text at scale +597%
All automated traffic +23.51%
Human traffic +3.10%

BARS ARE ON A COMPRESSED SCALE — A TRUE-TO-SCALE 7,851% BAR WOULD BE 330 METRES LONG AT THIS WIDTH. THE POINT IS THE ORDER OF MAGNITUDE, NOT THE PIXELS.

Training crawlers still account for 67.5% of all AI-driven traffic — the big, boring majority. More than 95% of AI-driven automation lands in just three verticals: retail and e-commerce, streaming and media, travel and hospitality. If you are in one of those, this is already your Tuesday.

Now the detail that breaks most published advice. That agent traffic is not arriving as a polite crawler with a documented user-agent. It is arriving inside a browser. In May 2026, agentic browsers made up the overwhelming majority of agent activity:

WHICH AGENT IS AT THE DOOR · SHARE OF AGENTIC TRAFFIC, MAY 2026 SOURCE: HUMAN SECURITY SATORI THREAT INTELLIGENCE
Overall agentic traffic actually fell 4.3% month over month in May — this is a young, jumpy market, not a smooth curve. Meanwhile the share of agent requests being blocked rose from 8.2% in April to nearly 9% in May. Roughly one agent in eleven is now bouncing off a door somewhere.

An agentic browser is a real browser. It runs JavaScript, holds cookies, carries the user's session, and looks — to most of your tooling — like a slightly odd Chrome. It is not fetching your robots.txt and asking permission, because it is not a crawler; it is a person's browser with a machine at the wheel. Every checklist that ends at "add these user-agents to robots.txt" stops one step short of the visitor that matters.

4. What it actually does once it's inside

Here is where the hype and the data part company, and this article is going to side with the data even though it makes for a smaller story. Agents are not swarming your checkout. In May 2026, three quarters of everything agents did on the web was looking:

WHERE AGENT REQUESTS LAND ON A SITE · MAY 2026 SOURCE: HUMAN SECURITY SATORI THREAT INTELLIGENCE
76.4%Product and search pages — comparing, reading, shortlisting
6.4%User account pages
5.6%Authentication — logging in
4.7%Content engagement — articles, guides, docs
2.4%Checkout and payment
By destination sector: media 43.4%, e-commerce 40.5%, travel 13.7%, financial services under 1% — but financial services grew 124% in a single month. The agent economy is arriving in the order you'd expect: read first, book second, pay last.

So: 2.4% is a rounding error, and anyone telling you that agents are about to become your main sales channel is selling something. But look at what 76.4% means. The agent is doing the shortlisting. It is the research phase — the exact phase where a business gets included or quietly dropped, three days before anyone talks to a human. You are not being bought from by a robot. You are being screened by one.

And screening is where you can still lose without ever knowing you were in the running.

5. They fail more often than the demos suggest

Every launch video shows an agent booking a flight in one clean pass. Then researchers put agents on real, live, unmodified websites, and the numbers collapse. Online-Mind2Web is the benchmark that did this honestly: 300 tasks across 136 live sites, graded by humans.

AGENT SUCCESS RATE · 300 TASKS ON 136 LIVE WEBSITES SOURCE: ONLINE-MIND2WEB (ARXIV 2504.01382)
What older static benchmarks claimed — WebVoyager~90%
Best agent tested on live sites — OpenAI Operator61.3%
Claude Computer Use 3.756.3%
SeeAct · Browser Use · Agent-E — the typical band28–31%
The authors' own tell: on the old benchmark a dumb search-only agent scored 51%, which means half the tasks had a shortcut. On the live benchmark the same dumb agent gets 22%. The gap between a demo and a Tuesday is roughly thirty percentage points.

Now put those two facts next to each other. Roughly two in five agent attempts fail on live websites, and roughly one in eleven agent requests is being blocked outright before it starts. Both of those failures are silent on your end.

A human who can't find your price gets annoyed, maybe calls. An agent that can't find your price returns to its user with two competitors and a sentence about how it couldn't confirm yours. You are not rejected. You are omitted, which is worse, because there is nothing to fix if you never learn it happened.

6. The part almost nobody tells you: it isn't looking at your page

This is the section to read twice, because it quietly rewrites what "a good website" means.

You probably assume an AI browsing agent takes a screenshot and looks at it, the way you would. Some do, some of the time. But the dominant, cheaper, faster and far more reliable method is to skip the pixels entirely and read the accessibility tree — the structured description of a page that browsers generate for screen readers. Microsoft's own Playwright MCP server, one of the most widely used pieces of agent plumbing in the world, states it in its README without a hint of ambiguity: it "uses Playwright's accessibility tree, not pixel-based input", and needs "no vision models" because it "operates purely on structured data".

So here is your homepage, twice. Left: what your designer signed off. Right: what a large share of agents actually receive.

THE SAME PAGE, TWO AUDIENCES RIGHT-HAND SIDE: AN ACCESSIBILITY SNAPSHOT, THE FORMAT AGENT TOOLING READS
01 · WHAT THE HUMAN SEES
Excellence, delivered. Your trusted partner since day one
GET STARTED

PRICE IS IN THE HERO IMAGE. PHONE NUMBER IS IN THE HEADER GRAPHIC. THE THREE SERVICE TILES ARE ICONS.

02 · WHAT THE AGENT RECEIVES
banner: img "" (no accessible name) main: heading "Excellence, delivered." [level 1] paragraph "Your trusted partner since day one" img "" (no accessible name) img "" (no accessible name) img "" (no accessible name) generic "GET STARTED" [not a button] contentinfo: img "" (no accessible name)

NO PRICE. NO PHONE. NO SERVICES. NO CLICKABLE CONTROL. FROM HERE THE AGENT CANNOT ANSWER A SINGLE QUESTION ABOUT THIS BUSINESS.

Illustrative, but not exaggerated — this is the shape of the snapshot agent tooling works from. Note what survives the trip: headings, link text, button labels, form labels, alt text, landmarks. Note what doesn't: anything that is only a picture, only a colour, only a position on screen, or only a div that happens to have a click handler.

Sit with the consequence, because it is genuinely good news. Everything that makes your site work for an agent is something that already makes it work for a blind customer. Real headings. Buttons that are buttons. Form fields with labels attached. Alt text that says what the picture says. Keyboard reachability. There is no new discipline to buy here — the discipline is thirty years old, it has a standard, and the industry has been half-ignoring it for a decade.

The agent-readiness product you were about to be sold is called accessibility, and it has been sitting in your backlog since 2019.

There's a second class of agent that never even gets a browser: the plain fetch tool inside an assistant, which pulls raw HTML and reads it as text. Anthropic documents that its web fetch tool does not support JavaScript-rendered pages at all. For that visitor, whatever your framework paints after load simply isn't there. Two very different machines, one shared requirement: put the facts in the markup.

A SMALL THING THAT HAPPENED WHILE WRITING THIS

Researching this article, my fetching tool asked HUMAN Security's own page about agentic traffic for its contents and got 403 Forbidden. A plain command-line request to the identical URL, in the same minute, got 200 OK. Nobody did anything wrong; that is just a bot policy doing its job, and their figures were readable elsewhere. But it is a perfect miniature of the whole problem: the door is answered differently depending on who knocks, the knocker gets no explanation, and the site owner never finds out.

7. The door test — seven checks, twenty minutes, no tools

None of this needs a vendor. Do it yourself this afternoon, on the three pages that actually earn you money.

THE DOOR TEST · WHAT TO CHECK, AND WHAT IT MEANS WHEN IT FAILS FREE · ~20 MIN · WORKS ON ANY SITE, ANY PLATFORM
1 Read your own robots.txt

Open yoursite.com/robots.txt and actually read it. You're looking for named AI user-agents sitting under a blanket Disallow: /.

IF IT FAILS: someone blocked the live fetch agents along with the scrapers. Two minutes to fix, and it may be the highest-value two minutes on this page.
2 Find the switch above robots.txt

Your CDN or WAF outranks your robots file. On Cloudflare, look for the AI traffic controls and check Search, Agent and Training separately. On other platforms, look for anything named "block AI bots".

IF IT FAILS: one toggle flipped in 2024 is still quietly turning away every customer's agent in 2026.
3 Turn JavaScript off and reload

Disable JS in your browser settings and open your most important page. Whatever remains is what a fetch-only agent gets.

IF IT FAILS: your prices, services and phone number don't exist for an entire class of assistant. Server-render the facts, even if the rest stays dynamic.
4 Do your main flow with the keyboard only

Hands off the mouse. Tab, Enter, Space. Get from your homepage to a submitted enquiry, booking or order.

IF IT FAILS: exactly where you get stuck is where the agent gets stuck — it is navigating the same tree your keyboard is.
5 Count the interstitials

Private window, homepage, ten seconds. Count everything that appears uninvited: cookie wall, newsletter modal, chat bubble, region picker, discount popup.

IF IT FAILS: every layer is a guess the agent has to make before it reaches a single fact about you. Modals interrupting mid-flow are a documented top cause of agent failure.
6 Try to select your own facts

Drag your cursor across your price, your phone number, your opening hours, your service area. If the text doesn't highlight, it's a picture.

IF IT FAILS: it isn't data, it's decoration. Nothing that lives only inside an image, a PDF or a canvas element makes the trip.
7 Give every control a real name

Buttons, links, form fields. "Book a call", "Email address", "Download the price list" are names. A bare icon, a div with a click handler, and a field labelled only by its placeholder are not.

IF IT FAILS: the control is invisible in the accessibility tree. The agent can see your form and cannot fill it.
8 Bonus: look at your own homepage the way a fetch tool does

One command, no install, macOS or Linux. It strips the tags and shows you the plain text an assistant receives. Read the first thirty lines out loud — if the answer to "what do you sell, where, for how much" isn't in there, that's your afternoon's work.

# what a fetch-only agent actually receives from your homepage curl -sL -A "Mozilla/5.0 (compatible; ChatGPT-User/1.0)" https://yoursite.com/ \ | sed -e '/<script/,/<\/script>/d' -e '/<style/,/<\/style>/d' \ -e 's/<[^>]*>/ /g' \ | tr -s ' \n' ' \n' | grep -v '^ *$' | head -30

If that command prints your headline, your services, a price and a phone number, you are already ahead of most of your competitors. If it prints "Loading…" or nothing at all, you have found the single most valuable thing on this page.

And the file that fixes half of it

If your robots.txt is a mess, this is a safe, explicit starting point. It lets in the machines that bring you customers, and leaves the training decision to you as an actual decision:

# allow the ones with a human waiting, and the ones that index you User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / # training crawlers — your call, and a real one. Uncomment to opt out. # User-agent: GPTBot # User-agent: ClaudeBot # User-agent: Google-Extended # Disallow: / Sitemap: https://yoursite.com/sitemap.xml

One trap worth knowing, because half-written robots files do more damage than missing ones: a crawler obeys only the group named after it and ignores User-agent: * entirely once such a group exists. Every rule you cared about has to be repeated inside the named group.

THE MACHINES WORTH KNOWING BY NAME · WHAT EACH ONE IS FOR PUBLISHED USER-AGENTS, AUGUST 2026
User-agentOperatorBehaviourWhat it costs you to block it
ChatGPT-UserOpenAIAgent · liveA person asked ChatGPT about you and it couldn't open your page.
Claude-UserAnthropicAgent · liveSame, in Claude. This is the one that costs actual jobs.
Perplexity-UserPerplexityAgent · liveSame, in Perplexity.
OAI-SearchBotOpenAISearch · indexYou're not retrievable in ChatGPT search, so you can't be recommended.
Claude-SearchBotAnthropicSearch · indexSame, in Claude.
PerplexityBotPerplexitySearch · indexSame, in Perplexity.
GooglebotGoogleSearch · indexYou leave Google Search. There is no version of this that is a good idea.
Google-ExtendedGoogleTrainingOpts you out of Gemini training and grounding. Does not remove you from AI Overviews.
GPTBotOpenAITrainingYour text stays out of future training data. Defensible. Costs little today.
ClaudeBotAnthropicTrainingSame decision, same trade.
Notice who isn't in this table: Comet, Atlas and Claude in Chrome — the browsers responsible for the overwhelming majority of agent activity. They arrive as a person's browser session, not as a named crawler, so no robots.txt line reaches them. For those, checks 3 to 7 above are the entire game.

8. What not to spend money on

⚠ FOUR THINGS BEING SOLD RIGHT NOW THAT DON'T WORK

"Agent-readiness certification." There is no registry, no submission form and no certificate that any assistant checks. The agent visits your live site and finds out for itself.

llms.txt. Eighteen months into the conversation, no major model vendor documents consuming an external llms.txt from your domain, and Google has publicly said it does not support it. We covered why it turned out to be a dead end. It costs ten minutes, so add it if you like — just don't count it as work done.

"GEO instead of SEO." Retrieval runs on a search index. Being findable by a search engine is not the old game you've graduated from; it is the entry ticket to the new one.

Blanket "block all AI" toggles. They were built when there was one kind of machine. There are three now, and the toggle can't tell your customer's proxy from a scraper.

9. The honest caveat, and why this still matters if you don't sell online

Let me argue against my own headline for a paragraph, because the strongest version of this case has to survive the objection.

Agentic traffic fell 4.3% month over month in May. Only 2.4% of it touches checkout. More than 95% of AI-driven automation is concentrated in three verticals — retail, media, travel — and if you run a specialist consultancy in Utrecht, you are not in any of them yet. The commerce protocols making the headlines, OpenAI's Agentic Commerce Protocol (September 2025, built with Stripe) and Google's Universal Commerce Protocol (announced at NRF on 11 January 2026, with Shopify, Etsy, Target and Walmart), are merchant infrastructure. If you don't have a cart, they are not your problem this year.

All true. And none of it changes what you should do on Monday, for one reason: every single item in the door test is something a human customer benefits from too. A page that states its price in text is better for the person on a train with bad signal. A form you can complete with a keyboard is better for the customer with a tremor. A homepage without four stacked popups is better for everyone who has ever visited a homepage.

This is the rarest shape a technology trend can take: the speculative future and the boring present ask for exactly the same work. You are not making a bet on agents. You are fixing your website, and getting agent-readiness for free.

If the agent thing turns out to be overhyped, you'll be left holding a faster, clearer, more accessible website. That is the worst case.

The short version

The majority of requests hitting the web are now machines, and most of them don't matter. One kind does: the agent acting live for a person who is trying to choose someone. It arrives inside a browser, reads your page as a structured tree rather than a picture, fails about two times in five, and never tells you when it failed.

You can't buy your way onto its shortlist. You can only be legible to it — and being legible to a machine turns out to be identical to being legible to a person in a hurry, in the dark, on a bad connection, with a screen reader. Twenty minutes, seven checks, no invoice.

Most of your competitors will read a headline about robots taking over the internet, feel vaguely alarmed, and change nothing. That is the whole opportunity, and like all such opportunities it has a shelf life.

HOW THIS ARTICLE WAS MADE

Every figure above is linked to its source below, and every source is primary or one hop from it: Cloudflare's own changelog for the three-category split, HUMAN Security's published benchmark for the traffic figures, the Online-Mind2Web paper for the success rates, and Microsoft's Playwright MCP README for the accessibility-tree quote. Where a number is soft, jumpy or one month old, this piece says so rather than rounding it into confidence. Landingpeak is not affiliated with Cloudflare, HUMAN Security, OpenAI, Anthropic, Google, Perplexity or Microsoft.