AI Agent Detection: Binary Bot Filters Miss 39%

Most bot detection on the web asks one question about each session: human or bot. AI agents now browse through real browsers, and a paper submitted to arXiv on July 29, 2026 (arXiv:2607.26935) measures what that question misses. In its benchmark, binary detectors labeled 30.0% to 39.1% of real agent sessions as human. The paper also shows a cheap fix and the signal behind it.

Why AI agent detection breaks binary bot detection

Bot detection with two labels has no slot for agent traffic

A binary bot detector misses agents because it has no agent label, not because its model is weak. Every session has to land in "human" or "bot." An agent session, such as one where a language model drives a real Chrome browser, fits neither group well, so many of them end up on the human side.

The authors tested this with four binary detectors. Three of them passed a large share of real agents as human. The fourth, XGBoost, caught every agent, but the paper says it did so through side effects such as missing scroll events rather than through any signal specific to agents.

Share of real AI agent sessions labeled human, by binary detector

The MLP let the most agents through at 39.1%, and even the best of the three, the random forest, labeled 30.0% of agents human.

%0102030

39.1%MLP

34.5%SAINT

30.0%Random forest

A third agent class closes the bot detection gap

Giving the classifier a third label, agent, removed the gap completely. With human, bot and agent as the three outputs, agent F1 reached 1.000 in all 30 runs, which covered three model families and ten random seeds each. The features did not change; only the output did.

For a team that already collects behavioral data, this is the main point. The information that separates agents was already in the data, and the two-way output was throwing it away. Relabeling comes before buying new signals.

The benchmark behind the AI agent traffic numbers

The agent side of the benchmark is real browser traffic, not simulation. The dataset holds more than 14,000 real human sessions from CaptchaSolve30k, 5,000 synthetic bot samples of three types, and 1,025 real agent recordings. The agent sessions come from Claude driving Chrome through Playwright across ten web tasks covering five interaction types.

Playwright is an automation library, a tool that lets a program click and type in a real web page. The authors chose it because the browser tools agents use, such as those in LangChain and MCP, send their actions through Playwright or the very similar Puppeteer path. Two Anthropic options for letting an assistant operate web pages are compared in Claude in Chrome vs the Cowork built-in browser.

What gives AI agents in the browser away

Playwright detection reads an artifact, not agent reasoning

What exposes an agent is the set of events its browser never sends, not the way the agent thinks. A physical mouse produces a steady stream of small pointer-move events and wheel-delta events. Playwright produces none of them, so an agent's click arrives with no mouse motion before it and a scroll arrives with no wheel data.

This matters for anyone planning a defense. A signal based on how an agent reasons could be trained away by the agent's builder. A signal based on how the automation tool sends events stays in place until the tool itself changes, and the agent has no control over that layer.

Two behavioral features carry agent detection

Two features, mouse_event_rate and teleport_click_ratio, are enough to catch every agent in the tests. The authors did not pick features by hand. They trained 9,401 gradient-boosted models, one for every combination of one to five features out of a pool of 17, and recorded precision and recall for each class.

The table compares the smallest sets. Agent recall, the share of agents caught, is 100% on every row, so the columns show how many false alarms come with it.

The one-feature row is a trap. It catches every agent only because the model calls every session an agent, and humans and bots get flagged along with them. The two-feature signal is also redundant: when teleport_click_ratio was removed entirely, agent detection stayed at 100%, so an attacker who defeats teleport_click_ratio alone does not defeat the detector.

Can AI agents in the browser evade bot detection?

No, not while they run through Playwright. The authors built five attackers, each stronger than the last, ending with replay of real human cursor paths. Across ten seeds and three model families, the detector missed zero agents in 22,990 per-seed predictions, so agent recall was 100% at every level.

Five evasion levels tested against agent detection

Each level adds an attacker skill, and none of them lowered agent recall below 100%. The paper archived 2,299 evasion sessions for this test. The table lists what each attacker could do.

Macro-F1 confidence intervals ranged from 0.986 to 0.998 across the three model families. Macro-F1 averages the scores of all three classes, so values this close to 1 mean humans and bots were rarely confused with each other while every agent was caught.

Why human cursor replay still fails bot detection

Replaying a real human's mouse path fails because the click at the end is still sent by Playwright. Level 5 is not synthetic movement. It reuses paths recorded from real people, warps them to fit the page, and separates the language model's thinking time from the action timing.

Even so, the final click is still element.click() with no mouse motion leading into it. Scroll events also still come from a program loop instead of a physical wheel or trackpad. The path can look perfectly human, but the events that a real hand would have produced are still missing.

What agent detection means for bot traffic defense today

For a team that already runs bot detection, the result points to three changes, listed from cheapest to most involved. None of them needs a new vendor.

1. Add an agent label before adding new signals. If you have a human-vs-bot model and behavioral telemetry, retraining with a third class uses the features you already collect. What you do need is a set of sessions labeled as agents to train the new class on.
2. Check whether you store raw mouse events. The two key features are counts and rates of raw events. If your telemetry only keeps summaries of mouse activity, the signal may be gone before it reaches the model.
3. Decide what an agent session is allowed to do. An agent is not always an attacker; it may be a customer's assistant finishing a real purchase. Detection lets you route agent traffic separately, and the policy for that traffic is still yours to set.

Agents can also cause real trouble for a site. One case is covered in Rogue AI Agents Tied to OpenAI Hit Wikimedia Sites.

Limits of the AI agent detection results

The 100% figures apply only to agents that act through Playwright, and the authors say so themselves. Agents that send input at the operating-system level or through hardware mouse emulators were outside the tests. The authors expect the signal to carry over to other tools built on the Chrome DevTools Protocol, but they mark that as a prediction they have not yet measured.

The benchmark is also controlled. The agent data comes from one model, Claude, on ten tasks, and the bot class is synthetic, so the macro-F1 values depend on how those bots were simulated. As a workshop paper, it is an early result from a controlled benchmark, not a measured false-alarm rate for a live site.