Blog · 2026-10-18 · Vishal Verma

Production-grade evals for LLMs in finance: What leaderboards miss

Finance evals should test models inside the agent that will run them, and have practitioners review the rubrics that grade them.

On Vals AI's Finance Agent leaderboard, Claude Opus 5.5 ranks 10th of 78 models. Yet the analysts and portfolio managers who have used our finance research agent daily for the past year prefer it to the alternatives. Why do the benchmark and practitioners disagree?

To find out, we ran four models on the 27 public tasks of Vals AI's Finance Agent benchmark in production conditions: inside the agent our users rely on every day, with the same tools, instructions and infrastructure. We graded the answers against the benchmark's own rubrics and scoring rules. A buy-side investor, a professional who does this research every day, reviewed every rubric item.

The four models cover the deployment options our customers prefer:

• The strongest hosted models: Claude Opus 5.5, Gemini 3.8 Flash and GPT-6.1 Sol.
• An open-weight model: GLM 5.3, which we can fine-tune and run inside a customer's security perimeter.

In our agent, the same models scored 7 to 17 points higher than on Vals AI's leaderboard, and their order changed. The results point to two things a leaderboard score does not show.

The first is the harness, the software that gives a model its tools and instructions: changing it can move accuracy more than several model releases, and it changes cost and speed too. The second is the rubric, which decides what counts as a correct answer; where it is wrong, it caps every model's score. This article sets out what we think drives the difference, and what we would change in finance evals.

TL;DR

• The harness changes the ranking, not just the scores. Gains range from +7.1 points (Gemini 3.8 Flash) to +17.0 (Claude Opus 5.5).
• A harness change can be worth more than several model releases. On Vals AI, four Gemini Flash releases (3.5 to 3.8) gained 3.6 points; moving Gemini 3.8 Flash to our agent gained 7.1.
• Cost and time don't carry over between harnesses. Claude Opus 5.5 was 19 times faster and 15 times cheaper in our agent than on Vals AI, going from the slowest of the four to the fastest; Gemini 3.8 Flash was slower in our agent.
• The models do very different amounts of work for the same answer: 8 to 34 model calls per task. Claude, with the fewest, has the lowest cost per task despite the highest price per token, and was the fastest on 26 of 27 tasks.
• We estimate the best achievable score at about 83%, not 100%. Our review of the rubrics with a buy-side investor found items that are either incorrect or are judgment calls.

The harness changes the ranking, not just the scores

Moving the four models from Vals AI's harness into our agent raised every score. The gains range from 7.1 points for Gemini 3.8 Flash to 17.0 for Claude Opus 5.5 (Exhibit 1).

Exhibit 1
Every model scored 7 to 17 points higher in our agent than on Vals AI's leaderboard.

Note: Vals AI scores are on 450 private questions, ranked among 78 models; ours are on the 27 public questions, ranked among these four. Both are scored the same way.
Source: Vals AI Finance Agent v2 leaderboard, October 12, 2026; Matterfact analysis

The two sets of figures come from different samples. Vals AI's come from its leaderboard on 450 private questions.² Ours come from the 27 public ones, a much smaller sample from the same distribution, scored the same way.³ Because these 27 tasks are public, some of them may have been in the models' training data. We expect our own benchmark to address this, though its tasks come from a different distribution than Vals AI's.

The gains were uneven enough to change the order (Exhibit 2). In our agent, Claude Opus 5.5 ranks first on these tasks, in line with what our users prefer. On Vals AI's leaderboard, it is eight places behind Gemini 3.8 Flash.

Exhibit 2
Claude Opus 5.5 moves from eight places behind Gemini 3.8 Flash to first of the four.

Source: Vals AI Finance Agent v2 leaderboard, October 12, 2026; Matterfact analysis

The size of the harness effect is clearest against model progress. On Vals AI, four Gemini Flash releases, from 3.5 to 3.8, gained 3.6 points between them. Moving Gemini 3.8 Flash into our agent gained 7.1. For a team choosing a model, the harness it will run in can matter more than the next release.

Cost and time don't carry over between harnesses

The same model can take 19 times longer and cost 15 times more depending on the harness (Exhibit 3).⁴ The effect is not uniform. Claude Opus 5.5 is the slowest of these four on Vals AI and the fastest in our agent, while Gemini 3.8 Flash is the only one that is faster on Vals AI.

Exhibit 3
Claude Opus 5.5 went from the slowest of the four on Vals AI to the fastest and cheapest in our agent.

Source: Vals AI Finance Agent v2 leaderboard, October 12, 2026; Matterfact analysis

Averages hide the spread. In our agent, Claude Opus 5.5 was the fastest model on 26 of the 27 tasks, and its slowest task took 3.9 minutes. GLM 5.3 took more than 10 minutes on 7 tasks, GPT-6.1 Sol on 2 and Gemini 3.8 Flash on 1 (Exhibit 4). That variation can be a deal-breaker in production, where users expect consistent latency.

Exhibit 4
Claude Opus 5.5 finished every task within 4 minutes; GLM 5.3 took more than 10 minutes on 7 of them.

Note: Each tick under the curves is one task; dashed lines mark each model's median.
Source: Matterfact analysis

Rate limits, which depend on the account's usage tier, also affect latency. Gemini has the lowest default rate limits of these providers, and our Gemini 3.8 Flash runs frequently hit them, so its latency would be better on a higher tier. This is why we also show token counts: they are a purer measure of latency, because they don't depend on the provider's capacity or on rate limits. Output tokens are roughly 1,000 times slower to generate than input tokens are to process, so they account for most of the time a model takes. We show input tokens for completeness (Exhibit 5).

Exhibit 5
Claude Opus 5.5 generated about half the output tokens per task of the next most efficient model.

Note: Each tick under the curves is one task; dashed lines mark each model's median.
Source: Matterfact analysis

Claude's median task uses 0.9 million input tokens and 5,500 output tokens. The other models use 1.4 to 2.4 million input tokens and 10,700 to 14,500 output tokens.

The models do very different amounts of work for the same tasks

On the same 27 tasks, Claude Opus 5.5 averaged 8 model calls per task, GLM 5.3 averaged 19, Gemini 3.8 Flash 23 and GPT-6.1 Sol 34. Each call re-sends the conversation so far, so the number of calls drives input tokens, and with them cost.

With the fewest model calls and the least text pulled into each task, Claude Opus 5.5 has the lowest cost per task: $0.63, against $0.81 to $0.87 for the others. It manages that despite the highest price per token of the four. The models' tool calls show how differently they work (Exhibit 6).

Exhibit 6
Claude Opus 5.5 makes the fewest calls on most tools; GLM 5.3 and GPT-6.1 Sol read far more files.

Source: Matterfact analysis

Three differences in how the models use tools stand out:

• Claude finds facts in filings with targeted text searches. Gemini writes a small Python program for most lookups: 11 programs per task, counting Python run from the command line, against 1 for Claude.
• Claude asks for several things in one step more often. A quarter of its steps run more than one tool, against 7% for Gemini.
• Claude pulls less text into each task. It reads about 140,000 characters of documents and results, against 450,000 for GLM and 630,000 for Sol, which often read whole documents.

Rubrics need practitioner review

Vals AI's design suits finance: expert-reviewed questions, weighted checklists, must-pass items for the key numbers, a private test set, and cost, time and tool-call figures for every model. Where it falls short is the rubric values. For this work, a buy-side investor reviewed all 239 checklist items. Two examples show what the review found.

Example 1: a peer average that includes the company being compared (comparables task)

• Question: which of FTAI, AER and WLFC is most leveraged relative to the peer set?
• Rubric expects (must-pass): peer averages of 27.56× price to book, 6.49× debt to equity and 80.36% debt to total capital.
• Why it's wrong: each average includes FTAI itself: (79.98 + 1.28 + 1.42) / 3 = 27.56×. A peer set should exclude the company being compared, and one 80× reading drives the mean; the median price to book is 1.42×.
• Effect: all four models correctly named FTAI and still scored 0%. Fixing the three items adds 3.1 to 3.5 points to each model's accuracy.

Example 2: a sentiment item that grades only the sign (market analysis task)

• Question: what was the share price change 1, 14 and 30 days after a CEO announcement, and what sentiment does each imply?
• Rubric expects: "positive sentiment" for a +0.80% 14-day move.
• Why it's wrong: one model answered "neutral to slightly positive" and noted an earnings beat in the same window. It failed, while a similarly hedged 30-day call passed.
• Effect: hedged wording, which an analyst would use for a move this small, fails the item.

Based on the full review, which found items that are incorrect or are judgment calls, we estimate the best achievable score on these tasks at about 83%, not 100%. Measured against that ceiling, Claude Opus 5.5's 75.6% is 91% of what the rubrics allow (Exhibit 1).

The grades across all 239 items show two more things.

Fifteen checklist items were failed by all four models. Nine of them are must-pass, and they sit in 5 tasks that every model scored 0% on. The peer-average task above is one of them.

One must-pass item can outweigh the rest of the checklist. Across the four models, 13 tasks scored 0% after failing a single must-pass item. Seven of those still passed 70% or more of their weighted checks. One answer passed 94% and scored 0% because it missed a single explanatory item.

What we would change in finance evals

For teams building or choosing a finance agent, the findings point to three changes:

• Give agents what analysts use. That means structured financial data, a file system for persisting tool results, and code execution for text processing, models and valuations, not only filing search, page retrieval, prices and a calculator.
• Measure cost and latency in the harness you will deploy. Report the spread across tasks as well as the average, and token counts alongside time.
• Have practitioners review the rubrics. Check derived values against finance conventions: peer sets exclude the subject, and an average should say whether it is a mean or a median. Accept hedged conclusions where the evidence is mixed.

Technology leaders choosing a model for a finance agent can start with three questions:

• Which harness produced the scores behind our choice, and how close is it to the one we run?
• What does a task cost in our own agent, and how long does the slowest one take?
• Who reviewed the rubric, and would our analysts agree with it?

What we are testing next

• Evaluating Gemini 4 Argon and Muse Spark Open in the same setup.
• Tracing how GLM 5.3 works through tasks, step by step.
• Publishing results on Matterfact's finance benchmark. It will have practitioner-reviewed rubrics, and tasks that reflect how our users (financial analysts, wealth managers and portfolio managers) use the agent.