Gemma 4 31B vs Sonnet 5: same score, 40x cheaper

What the 245-question finance benchmark measured

AlphaSense built the benchmark and ran it on its own search product, so the figures here are AlphaSense’s measurements, apart from a few sums and differences worked out from them. It is not an independent test. The 245 questions are modeled on real analyst workflows, and AlphaSense describes them as time-anchored, cross-source, and deliberately hard.

Company research and financial modeling lead the mix

The question set is sorted into eleven categories. Company Research is the largest block at 53 questions, and Financial Data & Modeling follows at 42.

The rest are smaller. General/Other and Earnings & Events have 24 each, and Deals & Due Diligence has 17. Sector, Industry & Macro has 15.

Competitive & Market Landscape, Expert & Primary Research, and Screening & Discovery have 14 each, and Regulatory, Policy & ESG and Coverage & Catalyst Monitoring have 13 each. The listed counts add up to 243, which is two short of 245.

Every model searched the same document set

Every configuration searched the same indexed documents, including the vector-RAG baseline and AlphaSense Search. Vector-RAG here is the baseline setup: raw vector search finds documents, and the model answers from them. Because the documents are identical, the test separates three variables: the retrieval, the harness (the software wrapped around the model), and the model.

Gemma 4 31B benchmarks: a tie with Sonnet 5 at 1.34

Gemma 4 31B and Claude Sonnet 5 sit on the same score, 1.34, in the middle of the AlphaSense score table. GPT-5.6 Sol is highest at 1.73, and Claude Opus 4.8 follows at 1.60. Claude Opus 5 scores 1.46, which is below Opus 4.8.

Nine models: five proprietary, four open-weight

The AlphaSense score table lists nine models. The five proprietary ones are GPT-5.6 Sol, Claude Opus 4.8, Claude Opus 5, Claude Sonnet 5, and Claude Haiku 4.5. The four open ones are Kimi K3, Gemma 4 31B, GLM-5.2, and Inkling.

AlphaSense score table: relative answer quality, all models on AlphaSense Search

Model	Weights	Relative answer quality
GPT-5.6 Sol	Proprietary	1.73
Claude Opus 4.8	Proprietary	1.60
Claude Opus 5	Proprietary	1.46
Kimi K3	Open	1.44
Claude Sonnet 5	Proprietary	1.34
Gemma 4 31B	Open	1.34
GLM-5.2	Open	1.31
Claude Haiku 4.5	Proprietary	1.30
Inkling	Open	1.29

Kimi K3 is the highest-scoring open model at 1.44, above Sonnet 5, and GLM-5.2 scores 1.31, which is 0.03 below the tie. Six of the nine models, from Kimi K3 down to Inkling, fall between 1.29 and 1.44, a spread of 0.15. Kimi K3 benchmarks, pricing and open weights and GLM 5.2 benchmarks, pricing, and Claude Code setup cover those two models.

Scores are relative to a 1.0 vector-RAG baseline

The scores are not accuracy percentages. Each is a ratio against the vector-RAG baseline, which is set to 1.0, so 1.34 means 1.34 times that baseline. The figures are medians from AlphaSense’s July 2026 harness run.

Gemma 4 31B cost: median per question, not list price

The roughly 40x figure compares what it costs Gemma 4 31B and Sonnet 5 to answer one question. It does not compare the price per token that each vendor publishes. AlphaSense notes that the costs in its chart are median per-question totals.

Retrieval-loop tokens count toward the total

A retrieval loop is the repeated searching and fetching a model does before it answers. AlphaSense includes those tokens in each model’s per-question total. Its own summary of the point is "Token price does not equal question cost."

Kimi K3 has cheap tokens; Opus 5 over-fetches

The notes in the AlphaSense score table give two examples. The note for Kimi K3 reads "Cheap tokens, but many of them." AlphaSense adds that Kimi K3 costs a lot less per token than GPT-5.6 Sol yet more per question, because it spends many more tokens assembling context before it answers.

Claude Opus 5 is marked "Over-fetches; most expensive per question." It still scores 1.46, third from the top.

Gemma 4 31B runs 10-20x faster on Cerebras

AlphaSense states that, because the weights are open, Gemma is served on Cerebras hardware, where the model runs 10-20x faster. AlphaSense reports this speed separately from the 40x cost figure. It does not say what the 10-20x speedup is measured against.

AlphaSense’s reading is that good-enough quality for many tasks, combined with very low latency, opens up high-volume and interactive uses that would be uneconomical on frontier models. It also calls Gemma a candidate for sub-agent delegation by a frontier lead model.

How the benchmark scores answers with frozen rubrics

A rubric is a checklist of what a good answer must contain. AlphaSense gives each of the 245 questions its own rubric. Graders then score the answers against it.

Rubrics are written before any model answers

AlphaSense generates each rubric before any model produces an answer, and the rubric is frozen once it exists. It names two reasons.

Writing the criteria first stops graders from drifting toward whatever the models happen to output. It also lets AlphaSense encode the hard requirements up front.

Rubrics grade evidence, with weights from -5 to +5

A rubric grades the evidence, not the prose of the answer. Each criterion has to be checkable from the retrieved documents alone, and each criterion is assessed on seven axes.

The size of a rubric scales with the question. A pinpoint lookup gets 3-5 criteria, a bounded comparison gets 10-16, and a multi-hop landscape gets 18-30. A question can have anywhere from 1 to 60 criteria.

Each criterion has an integer weight from -5 to +5, and never 0. Roughly 40% of the positive weight goes to temporality, authority, and breadth.

Open weight models vs Claude: routing beat any single model

In AlphaSense’s words, no model wins more than a plurality of the benchmark questions. The routing results below are about which model each kind of question is sent to. AlphaSense compares them with GPT-5.6 Sol, the best single model.

Different models win on different question types

According to AlphaSense, different models win on different question types: metric extraction, screening, and long-form synthesis. AlphaSense describes model routing as more than a way to save money.

The aim is the best model for each use case, with an optimal combination of quality, cost, and speed for each task. The same idea is sometimes called LLM routing.

Model routing reaches 2.8:1 versus 1.8:1 for one model

AlphaSense reports three preference ratios in sequence. Choosing the best frontier model, GPT-5.6 Sol, on top of AlphaSense retrieval pushes preference to 1.8:1. Adding the optimized AlphaSense harness around that model (query generation, planning, and search score pass-through) gets to 2.1:1, a gain of 0.3.

Routing each task to the best model for that task type reaches 2.8:1, the highest of the three. That is 1.0 above the 1.8:1 for GPT-5.6 Sol on its own.

Scope of the result: finance questions on AlphaSense Search

What the benchmark covers

Item	What AlphaSense measured
Domain	245 finance research questions
Quality metric	Relative answer quality, with the vector-RAG baseline at 1.0
Cost metric	Median total per question, including retrieval-loop tokens
Run	July 2026 harness run, all models on AlphaSense Search

Everything ran inside AlphaSense’s own search

All nine models ran on AlphaSense Search, so the scores describe that system. AlphaSense calls the gaps it reports a floor and says they measure only the search agent.

AlphaSense reports a 3x retrieval cost gap

AlphaSense also reports what happens when a frontier model does its own retrieval with raw vector search. That setup costs roughly 3x more per question than the same model in AlphaSense’s harness. With the same model and corpus, the harness costs about a third as much and gives significantly better answers.

95% accuracy per step compounds across ten steps

AlphaSense says the quality and cost effects compound in longer-running agentic workflows. In its example, even if 95% accuracy is acceptable in a single step with human oversight, it compounds to coin-toss accuracy across a 10-step agentic process.