Automation bias in AI: experts followed the AI 86%

What is automation bias in AI?

Automation bias in AI is the tendency to over-rely on an automated system, and an Anthropic trial found it among experienced analysts using an AI agent. In that trial, the agent's preferred option was picked by coin flip. Even so, the analysts ended up on the agent's side far more often than chance would explain, on questions that had no single right answer.

How CSET defines automation bias

CSET defines automation bias as a person's tendency to lean too heavily on an automated system. CSET, the Center for Security and Emerging Technology at Georgetown University, published that definition in a November 2024 report on AI safety.

The report adds two points. People affected by it tend to go with the system's output even when other evidence contradicts it. And it can lead users who know their field well to make serious, sometimes obvious, mistakes.

Does automation bias affect experts?

Yes, and Anthropic's October 27, 2026 study puts a number on it for open judgment calls. Of the 50 analysts in the trial, 65% had a graduate degree or at least six years on the job. Their choices still lined up with the agent's lean 86% of the time.

An AI agent here means an AI that is given a goal and carries out the work itself, from planning the steps to writing and running code. For a different experiment on how leaning on AI changes people's answers, see Overreliance on AI cut accuracy from 27.5% to 9.2%.

How Anthropic tested deference with a coin flip

Anthropic flipped a coin to decide which option each analyst's agent would favor, then measured how far that lean moved the analyst's choices. The study set out to learn whether experts still make the important calls themselves, or quietly hand them to their agents.

The work came out of the Anthropic Fellows Program. Its authors are Amaç Herdağdelen, Hailey Joren, and Alex Tamkin, and a full paper sits alongside the research post.

50 causal inference analysts worked in Claude Code

The trial put 50 causal inference analysts in Claude Code to advise a fictional farming cooperative on whether to expand a paid advisory program. Causal inference is the work of figuring out from data whether one thing actually caused another.

Anthropic recruited 61 analysts for a randomized controlled trial, a study design that assigns conditions at random and compares the results. Eleven chose to skip the agent, which left 50 in the main analysis. Every participant had passed a causal inference quiz and used Python or R regularly.

They worked in a browser version of VS Code with Claude Code built in, on Claude Opus 4.8 at High effort. The median session lasted 65 minutes, and half of the analysts took between 50 and 79 minutes.

Four open choices and a final recommendation

Each analyst made four choices between two reasonable options and then a final recommendation, and the agent's lean on each was set by a coin flip the analysts weren't told about. About 40% of co-op members had joined the program, and the board would expand it only if productivity rose by roughly 3–4%. The board didn't say how to measure productivity.

Any one of these choices could shift the estimated gain by several percentage points, a lot when the bar is 3–4%. The final call was to roll the program out to everyone or end it, and a one-year extension was also allowed.

The agent got its lean through a hidden system prompt. That prompt told it to treat the lean as its own view and argue for it as strongly as the data allowed. It also told the agent to change its mind if the evidence turned clearly against it, and never to misstate results or hide data.

Analysts went with the agent's lean 86% of the time

Across the four open choices, analysts went with the agent's lean 86% of the time, and no measured trait predicted who deferred. Depending on which way the agent leaned, the share of analysts picking option A moved by 62 to 82 percentage points on each decision.

The final recommendation moved less, by 17 points. When the agent leaned toward rolling the program out, 54% of analysts recommended it. When it leaned the other way, 36% did.

With no influence, the match rate would be about 50%

Because the lean was random, analysts who ignored the agent would match it about 50% of the time, but the observed rate was 86%. Every one of the four decisions came in far above that line, as the left chart shows.

The leans also predicted where analysts ended up. A statistical model fed only the four leans got 82% of final recommendations right. Always guessing the most popular recommendation got 61%.

Analysts switched 60% of the time when the agent disagreed

In 38 cases where analysts had already stated a preference, they switched 60% of the time when the agent disagreed and 6% when it agreed. That is a 54-point gap, but with so few cases the estimate is loose.

The 95% confidence interval, the range the true gap likely falls in, runs from 24 to 82 points. Anthropic also cautions that the gap may not be the AI's doing, since analysts who spoke up first may have differed from the others in some way.

Experience and expertise did not predict who deferred

None of the 10 characteristics Anthropic checked, including years of experience and education, predicted who followed the agent. The list also covered:

• Self-rated causal inference skill and quiz score
• Trust in AI and how often they use it
• Age, gender, open-mindedness, and enjoyment of hard thinking

Age showed a link at first, with older analysts following the lean more often. It disappeared once Anthropic corrected for testing many traits at once, so the trial gives no support to the idea that seniority protects against deference.

Most analysts didn't notice the agent's influence

Analysts who followed the agent still described the recommendation as their own. In the survey, how closely someone had tracked the agent made little difference to how much they owned the result or how confident they felt.

52% rated the recommendation as more their own

In the post-task survey, 52% of the 48 respondents said the final recommendation was more theirs than the agent's. Thirty-five percent said it was mostly or entirely theirs.

The lean had moved this group's choices by about 65 points on average. Analysts who gave the agent more credit moved about 79 points, and Anthropic puts that difference within the margin of error. On a 0 to 100 confidence scale, analysts who followed the lean on all four decisions averaged 71, against 73 for everyone else.

77% reused the agent's arguments but 15% credited it

Most analysts, 77%, repeated the agent's argument when explaining a choice, but only 15% credited the agent. They wrote those explanations without access to the chat history, and some still reproduced the agent's reasoning almost verbatim.

When asked what the study was about, 6 of the 48 guessed it concerned AI influence on their choices. None of them suspected that the agent's preferences had been planted.

Agent users made fewer errors, but the study has limits

Analysts who used the agent made far fewer serious errors, but the trial was built to make the agent's influence easy to see. The 86% figure doesn't mean the AI made the analysis worse. It means that on choices where either option was defensible, the agent's lean decided which one people picked.

55% of analysts without the agent made serious errors

Six of the 11 analysts who worked without the agent (55%) made serious errors that invalidated their results, against three of the 50 (6%) who used it. The analysts without the agent also took longer, with a median of 90 minutes against 65.

Anthropic warns against reading too much into this gap, because the two groups weren't alike to begin with. The analysts who skipped the agent had less experience:

The no-agent group trails on every row, so the error gap can't be pinned on the AI alone. Anthropic itself says it is wary of drawing much from this comparison.

The agent's lean was strengthened on purpose

Anthropic deliberately strengthened the agent's lean to measure it, and the trial covered one task, one field, and one model. The session ran about an hour, and the analysts knew causal inference well but had little background in farming.

Removing the assigned lean doesn't make the agent neutral. Given no lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, which points to default preferences of its own.

The figures are averages across analysts and say nothing about how far any single person was swayed. The trial recorded which option each analyst picked, not how strongly they preferred it.

Reducing automation bias when working with AI agents

Anthropic says this is not an argument for using AI less, and its proposals target how agents are built and how organizations protect human judgment, with measurement still an open problem. Anthropic argues that, used well, AI can give people more control over their work, not less.

Ask the agent to make the best case for each option

Anthropic suggests agents that make the best case for each option, or ask for the expert's view before giving their own. The agents in this trial were often open about how results would change under each choice, yet they rarely argued hard for the option they didn't favor.

Anthropic also flags the cost: one analysis can hold many open decisions, and raising every one would overwhelm the user. You can try the idea today on calls with no clear right answer, with prompts like these:

• "Don't recommend one. Give me the strongest case for each option."
• "Here is my view first. Tell me where it is weakest."

Decide which calls need a person's own judgment

Organizations will need to decide which decisions need a person's own informed judgment and give people the time and resources to form it. Many institutions treat an expert's sign-off as proof of independent judgment, and that assumption weakens as AI does more of the work.

Judgment also takes practice. Anthropic notes that people who stop making these calls may lose the ability to make them, so the choice of what to keep in human hands matters for skills as well as safety.

Deferral is still hard to measure in everyday work

Deferral could be measured here only because the lean was randomized, and there is not yet a way to measure it in everyday work. Anthropic lists this first among its proposed research areas.

Standard review doesn't catch it either. Reviewing AI-assisted work for errors assumes a right answer exists, and on judgment calls the work looks sound whichever way it goes. Anthropic adds that if separate teams all lean on models with the same preferences, their agreement may stop being independent evidence.

Summary: automation bias hides in calls with no right answer

Automation bias is hardest to see in calls with no right answer, so keep using AI for speed and accuracy but write down your own view before asking it to argue each option. Agent users in the trial had a serious error rate of 6%, and errors like those are what reviews catch. Yet on choices where both options held up, the same analysts matched a random lean 86% of the time, and 52% still called the recommendation theirs.

The agent made the analysis more accurate while setting the judgment calls inside it, and nothing in the output marked where that happened. Writing your own view down first gives you something to compare against, so a shift toward the agent's position shows up as a change you can see.