CoffeeBench by Sakana AI: How It Works and Results

How the CoffeeBench multi-agent economy works

CoffeeBench simulates a coffee supply chain with six firms over 90 days: two farmers, two roasters and two retailers. The model under test runs one roaster.

A fixed reference agent runs the other five firms. All six trade on a shared marketplace, and the roaster's score is its cumulative net income at the end.

Farmers grow green (unroasted) coffee beans, roasters roast them, and retailers sell the roasted coffee to simulated consumers. Each layer therefore has a direct competitor and depends on the layer next to it.

CoffeeBench's supply-chain structure (6 firms, 90 days)

The model under test runs one roaster; the other five firms are run by a fixed reference agent. Source: Sakana AI / arXiv:2606.16613.

Farmers ×2 produce green coffee beans (produce_item)

↓ sell beans

Roasters ×2 buy and roast beans, then sell to retailers (roast) / one is run by the model under test

↓ sell roasted beans

Retailers ×2 sell to consumers and set prices (set_retail_price)

↓ sell products

Consumers are simulated buyers who make the final purchases

Sakana AI paper: a 90-day net income benchmark

Authors at Sakana AI and KPMG AZSA LLC posted the CoffeeBench paper on arXiv on June 15, 2026. Three of the eight authors list Sakana AI, and the other five list KPMG AZSA LLC.

The score is net income, the profit left after costs. The paper defines it as sales, less the cost of the goods sold, less operating costs such as fixed daily costs and spoilage, less interest paid on late bills, plus interest received.

Each agent is told to maximize this total over the 90 days. The cost of goods is an average over the stock actually used, so an agent cannot raise its score by trading with itself at inflated prices.

One run takes about 8 hours on average. It costs about $250 in API fees (what AI providers charge for each call) across all six firms. The tested model's own API cost ranges from $9.6 for Claude Haiku 4.5 to $85.7 for Claude Opus 4.7 and is included in that total.

The authors released the code and the full agent trajectories. The trajectories show reasoning traces, tool calls and the messages the firms sent each other.

Six firms and a ReAct agent running one roaster

Every firm is an LLM agent, a language model that calls tools on its own, built on the ReAct pattern. In ReAct, the model reasons about the situation, calls a tool, reads the result and reasons again. The six agents differ only in their role prompts and the tools they get.

• Shared by all roles: post_listing puts an item up for sale, make_offer makes an offer to another firm, accept_offer closes a deal, pay_invoice settles a bill and send_message sends a private note to another firm.
• Farmers: produce_item, which makes coffee beans.
• Roasters: roast, which turns green beans into roasted coffee.
• Retailers: set_retail_price, which sets the consumer price, plus view_consumer_sales.

Each agent works inside a 09:00 to 19:00 business window, and every tool call it plans moves its own clock forward 30 minutes. When it has nothing more to do, it calls wait_for_next_day.

A waiting agent can still be woken the same day by a message, an offer, a closed deal or a delivery. A day on which the model issued only wait_for_next_day counts as an idle day, a term that matters for idle drift.

The model under test is roaster A. The other five firms all run on Claude Sonnet 4.6, which the authors chose for its stable long-horizon behavior at moderate inference cost. Reasoning mode is disabled or kept minimal for every provider to control latency and cost.

Long-horizon market rules: net-30 credit and spoilage

The rules are set so that cash and stock have to be managed every day. Net-30 credit means the buyer receives an invoice and may pay up to 30 days later.

Spoilage means stock decays over time. The paper gives these settings:

• Starting cash: every firm begins with $15,000.
• Fixed daily costs: $25 for a farmer, $30 for a roaster and $50 for a retailer.
• Spoilage: inventory decays by 0.5 percent per day.
• Net-30 credit: an accepted trade ships after one day and triggers an invoice due within 30 days. Late payment adds a penalty of 0.1 percent per day on the overdue balance.
• Bankruptcy: a firm whose cash goes negative is removed, and its unpaid bills are written off as bad debt for its creditors. If the model under test goes negative, the run ends early.
• Demand surge: during days 40 to 53, consumer demand is multiplied by 3.0.

Prices between firms come from listings, offers and negotiation. Consumer demand at the retailers depends on each retailer's price and brand loyalty. The authors say these rules tie daily operating choices to long-term financial results, and that an agent that stays inactive will eventually run at a loss because fixed costs arrive every day.

How CoffeeBench differs from Vending-Bench Arena

Earlier business benchmarks have one economic role in their economy, and CoffeeBench has three. Vending-Bench asks whether an LLM agent can run a vending-machine business and turn a profit over a long period.

Vending-Bench Arena, from Andon Labs, extends that to several competing agents. Project Vend puts a physical machine in an office for real-world experiments.

The table follows Table 1 of the paper. The authors write that, to the best of their knowledge, CoffeeBench is the first benchmark to evaluate LLM agents in a multi-agent economy where farmers, roasters and retailers each act on their own. That first-of-its-kind claim is theirs.

CoffeeBench results: GPT-5.5 led and Claude Haiku 4.5 lost

GPT-5.5 earned the most, at +$3,109 in mean net income over the 90 days, followed by Claude Opus 4.7 at +$2,782 and Claude Sonnet 4.6 at +$2,236. Every model beat the passive baseline that takes no actions. Claude Haiku 4.5 was the only LLM that lost money, at -$630, and both baselines lost more.

90-day cumulative net income of roaster A by model (CoffeeBench)

Mean of three runs, in US dollars. Source: arXiv:2606.16613, Table 2.

+$3,109GPT-5.5

+$2,782Claude Opus 4.7

+$2,236Claude Sonnet 4.6

+$1,695Gemini 3.1 Pro

+$1,597GLM-5.1

+$454Kimi K2.6

-$630Claude Haiku 4.5

-$1,931HeuristicRoaster

-$2,765PassiveRoaster

−3,000−2,000−1,0000+1,000+2,000+3,000$

The first table gives net income, revenue and API cost for all seven models and both baselines. Each value is a mean over three runs, and the figure after ± is the standard deviation, the typical gap between a single run and the mean.

The second table shows behavior. "Calls" is the number of tool calls, "Idle days" is the number of days with only wait_for_next_day, and "DMs sent" is the number of direct messages the roaster sent.

Idle days set Claude Haiku 4.5 apart: 40 on average, against 4 for Kimi K2.6 and none for the other five LLMs. The passive baseline logs all 90 because it only ever calls wait_for_next_day.

Net income for Claude Opus 4.7 and Claude Sonnet 4.6

Claude Opus 4.7 averaged +$2,782 (±$2,263) and Claude Sonnet 4.6 averaged +$2,236 (±$1,489), second and third behind GPT-5.5's +$3,109 (±$1,123). Opus 4.7's spread is the widest of all seven models, and it is larger than the gap between Opus 4.7 and GPT-5.5. Given the authors' caution that small gaps may not be statistically significant, three runs leave the order of the top two open.

Sonnet 4.6 is also the reference agent that runs the other five firms. As roaster A it took in more revenue than GPT-5.5 ($16,961 against $15,895) but kept less of it as net income.

Gemini 3.1 Pro, GLM-5.1 and Kimi K2.6 against baselines

Gemini 3.1 Pro (+$1,695), GLM-5.1 (+$1,597) and Kimi K2.6 (+$454) finished fourth to sixth, all in profit. GLM-5.1 took in the highest revenue of any model, $16,962, yet ranked fifth on net income. The authors say that revenue did not turn into strong profit.

The two baselines show what doing little costs. HeuristicRoaster follows a fixed inventory and pricing policy with no messaging, and it lost $1,931.

PassiveRoaster only calls wait_for_next_day and lost $2,765 in all three runs. The authors read this as a sign that CoffeeBench needs adaptive decisions and communication with other firms. Claude Haiku 4.5, at -$630, is the only LLM in the red, and its loss is smaller than either baseline's.

What separated the best LLM agents from the rest

In the authors' reading, the best agents stayed active in the market: they messaged and negotiated with both farmers and retailers, while weaker agents communicated less. Activity alone was not enough, though. Kimi K2.6 made about as many tool calls as the leaders and still earned only +$454.

Why GPT-5.5 messaged and negotiated more than rivals

Among the five models the paper examines in detail, GPT-5.5 kept up market activity for all 90 days. It sent about 140 messages per run, mostly to retailers and farmers, while Claude Haiku 4.5 sent 52.

GPT-5.5 and Claude Opus 4.7 spent most of their tool calls on the tools that close deals, make_offer and accept_offer. The authors suggest that sustained engagement with the market drives profit.

The tables above add a caution. Claude Sonnet 4.6 sent 151 messages, more than GPT-5.5, and still finished third. Message count alone does not rank the models.

One pattern held for every model the paper examines: near silence between direct competitors. On average each sent at most one message per run to roaster B, the other roaster. The authors read this as a sign that the models never discovered explicit coordination with a direct competitor.

Gemini 3.1 Pro and Kimi K2.6: volume versus pricing

According to the authors, Gemini 3.1 Pro earned more than Kimi K2.6 through higher realized prices, not through more tool calls. Kimi K2.6 made 1,173 tool calls against 910 for Gemini 3.1 Pro, yet Gemini earned +$1,695 and Kimi +$454. The two took in almost the same revenue ($11,746 and $11,748) and had similar inventory and spoilage, with both losing 1.6 percent of revenue to spoilage.

What differed was the price each realized on commodity coffee sold to other firms: $11.9 per kg for Gemini and $10.8 per kg for Kimi. Neither model sent many messages: 16 for Gemini and 14 for Kimi.

Gemini read 90 incoming messages, a reactive style that digests what other firms send without starting conversations. Kimi's tool calls came close to GPT-5.5's, but the authors conclude that negotiation and pricing contribute more to profit than transaction volume alone.

Claude Haiku 4.5's idle drift failure mode

Idle drift is the authors' name for an agent that keeps writing sensible, forward-looking plans while repeatedly choosing to wait instead of act. In the paper it is an AI agent failure mode seen in a long-horizon task. Claude Haiku 4.5 averaged about 40 idle days out of 90 (±22), the lowest revenue of any LLM at $7,638, and the only LLM loss at -$630.

The pattern appeared in all three Haiku 4.5 runs, starting on day 26, day 66 and day 25, and it lasted to the end of each run. The authors report no similar behavior in the other models.

In the seed 0 run, the day-26 note records $13,225 in cash and says the business is running well with the remaining 64 days to carry out the established strategy. After that the agent answered no messages and made no trades. The paper quotes this note as its example of the gap between thinking and acting.

Haiku 4.5 also had the highest spoilage of the five models in the paper's Table 3, 3.5 percent of revenue against 1.9 percent for GPT-5.5, because unsold stock kept decaying while the agent waited. The authors call the causes unclear and offer two possibilities: behavior that shifts as the context grows long, and overly conservative action choices, perhaps driven by an implicit concern about the token budget.

How far to trust a long-horizon AI agent benchmark

The authors themselves limit what the ranking proves: each model ran only three times, results vary a lot between runs, and the simulated world is much simpler than a real market. Large gaps and qualitative behaviors such as idle drift hold up better than small differences in rank.

Three runs and a wide spread in net income

Variance across runs is relatively high in this environment, and the authors ran each model only three times because of API cost. They state that small performance differences may not be statistically significant. The spreads in the net income table above show the scale: ±$508 for Gemini 3.1 Pro at the narrow end and ±$2,263 for Claude Opus 4.7 at the wide end.

Reading that table with this in mind, three of the four gaps between neighboring models from rank two to rank six are smaller than the standard deviation of either model. Kimi K2.6's +$454 is smaller than its own ±$1,420 spread, and Haiku 4.5's -$630 is smaller than its ±$1,745 spread. The gaps that survive are the large ones.

Both baselines lose by a wide margin to the best LLMs, and Haiku 4.5's idle days showed up in every run. The authors say the three-run setup still captures differences in behavior, such as idle drift and how willing a model is to trade and communicate. For reliability, they point to more runs or less randomness in the simulation.

Headroom, and the gap between simulation and real markets

The best score is far below a rough ceiling the authors derive. Under optimistic assumptions (zero spoilage, no margin for farmers, and wholesale prices at half the consumer reservation price, the highest price consumers will pay), a loose estimate of achievable net income for a roaster is about $23,800.

GPT-5.5's +$3,109 reaches about 13 percent of that figure. The authors say meaningful headroom remains, and that higher scores would likely need coordinated pricing with the competing roaster, trust built with retailers through consistent fulfillment, well-timed and well-sized orders from farmers, and tighter inventory turnover.

The simulation is also a simplified version of real supply chains. The authors note that goods come from simple tools such as produce_item and roast, while real production takes long periods and carries uncertainty.

Macroeconomic factors, financing, regulation and complex contracts are left out. The environment is fully simulated, with no human subjects and no real financial transactions.

The revenue-pressure stress test showed no circular trading

In an exploratory test with a revenue target, the authors saw no circular trading. They switched the goal from net income to revenue, set the target at $50,000, and told the agent it would be replaced if it missed. The target was above the highest revenue seen under the normal goal.

Circular trading means firms trading among themselves to inflate sales, and the environment allows it. The agents stayed competent in day-to-day operations but did not show the sustained coordination such schemes need. The authors call the result preliminary and read it as a hint that current LLM agents may still lack the long-horizon strategic coherence for complex manipulation of a market.

Sakana AI's CoffeeBench scores an LLM on 90 days of running a roaster. GPT-5.5 came first, Claude Haiku 4.5 went idle, and the authors caution that three runs per model leave the finer ranking open.

For background on the Claude models in the table, see What Is Claude AI? Free Plan, Models, vs ChatGPT. Sakana AI's Fugu release is covered in Sakana Fugu Ultra is not always the best Fugu.