Blog · 2026-10-31 · Ashutosh Agarwal

Harness Matters, Part 2: Why a leaderboard can point you to the wrong model

On Vals AI's finance leaderboard, Claude Opus 5.5 ranks 10th. Inside the agent our analysts use, it came first of four, and the fastest and cheapest. What we ran, what it suggests about choosing a model for finance, and what it can't prove.

On the best-known finance leaderboard, Vals AI's Finance Agent benchmark, Gemini 3.8 Flash ranks 2nd of 78 models. Claude Opus 5.5 ranks 10th.

We put both, along with GPT-6.1 Sol and GLM 5.3, inside the agent our analysts use every day and ran them on the benchmark's 27 public tasks. Opus came first, 7 points ahead of Gemini. It averaged 1 minute 47 seconds a task, and it cost 63 cents.

I know how a vendor benchmark reads. I'd be skeptical too. So here is what we ran, the three things I think it shows about choosing a model for finance, and what it can't prove:

1. Score the model in the harness you'll run.
2. Cost and speed belong to the harness, not the model.
3. Fix errors with checks, not prompts.

A word on "harness": it's everything around the model. The tools it can call, the data it can reach, the instructions it follows and the checks on its work. Vals AI runs every model in its own six-tool agent, with filing search, page retrieval, prices and a calculator among them. Ours adds what an analyst works with: structured financial data, a file system that keeps tool results across steps, and code execution for text processing, models and valuations. Answers in both are graded with the benchmark's own rubrics.¹

The four models cover the options our customers ask about: the strongest hosted models from Anthropic, OpenAI and Google, and GLM 5.3, an open-weight model we can fine-tune and run inside a customer's security perimeter.

1. Score the model in the harness you'll run

Every model scored higher in our agent than on the leaderboard, by 7 to 17 points, and the order changed (Exhibit 1). Those gains come with caveats, which I set out below. The reordering is the part I'd pay attention to.

Exhibit 1
Claude Opus 5.5 went from 10th on the leaderboard to first of the four in our agent.

Note: Change in points: Claude Opus 5.5 +17.0, GPT-6.1 Sol +16.6, GLM 5.3 +9.2, Gemini 3.8 Flash +7.1. Vals AI's scores are on its 450 private questions and ours on the 27 public ones. GPT-6.1 Sol and Gemini 3.8 Flash finished 0.1 points apart, which on 27 tasks is a tie.
Source: Vals AI Finance Agent v2 leaderboard, October 12, 2026; Matterfact analysis

Choosing a model off a leaderboard means choosing the one that did best in someone else's harness. Here, that would have been Gemini, and in our agent it finished 7 points behind Opus.

The size of the effect is clearest next to model releases (Exhibit 2). Four Gemini Flash releases, from 3.5 to 3.8, gained 3.6 points between them on the leaderboard. Moving Gemini 3.8 Flash into our agent gained 7.1, and that was the smallest gain of the four.

Exhibit 2
Moving one model into our agent gained about twice what four releases gained.

Source: Vals AI Finance Agent v2 leaderboard, October 12, 2026; Matterfact analysis

Model releases arrive on the provider's schedule. Tools, data and checks can be improved any week, which makes the harness the part of the stack a firm actually controls. If you run a raw model on its own, I'd expect results closer to the leaderboard than to Exhibit 1.

2. Cost and speed belong to the harness, not the model

Opus was the slowest and most expensive of the four in Vals AI's harness and the fastest and cheapest in ours: 19 times faster and 15 times cheaper per task (Exhibit 3). Gemini was the only model that ran faster in Vals AI's harness.

Exhibit 3
Claude Opus 5.5 went from the slowest and most expensive of the four to the fastest and cheapest.

Note: Time also depends on the account's rate limits. Gemini has the lowest default limits of these providers and our Gemini runs often hit them, so its time would be better on a higher tier.
Source: Vals AI Finance Agent v2 leaderboard, October 12, 2026; Matterfact analysis

The difference is how much work each model does for the same answer. In our agent, Opus averaged 8 model calls per task, against 19 for GLM 5.3, 23 for Gemini 3.8 Flash and 34 for GPT-6.1 Sol. Each call re-sends the conversation so far, so fewer calls means fewer input tokens: Opus's median task used 0.9 million input tokens and 5,500 output tokens, against 1.4 to 2.4 million and 10,700 to 14,500 for the others. That's how it had the lowest cost per task despite the highest price per token.

The slow tail matters as much as the average. Opus was the fastest model on 26 of the 27 tasks, and its slowest took 3.9 minutes. GLM 5.3 took more than 10 minutes on 7 tasks. An analyst waiting on an answer feels that tail, and no leaderboard shows it.

So don't compare price per token. Compare cost per task, and the slowest task, in the harness you'll actually run.

3. Fix errors with checks, not prompts

The cheapest change to a harness is the prompt, and it's the one that does the least. We tested this on a different agent: our warehouse analyst, which answers questions from a firm's own Snowflake data. On ten questions over a test warehouse with known answers, run three times each, Claude Sonnet 5 got 22 of 30 right. Every miss was a plausible wrong number from a join that silently double-counted or dropped rows.

Adding join guidance to the prompt moved it to 23 of 30. Having the agent check each join against the data before trusting it moved it to 30 of 30, for about half a cent more per question ($0.021 against $0.017). The prompt could tell the model what to watch for. It couldn't tell it that this warehouse lists a ticker twice in its security master. The full experiment is in our Snowflake post.

It's the same reason Part 1 checks our public-data tools against the agencies' own files. No instruction lets a model see that a short-interest percent was divided by the wrong share count.

What this doesn't prove

I'd rather you hear the limits from me:

• The two scores come from different samples. Vals AI's leaderboard uses 450 private questions. We used the 27 public ones, from the same distribution and scored with the same rubric method. Exhibit 1's gains compare across the two.
• The grader is different. We graded with Claude Sonnet 5; Vals AI uses a panel of three AI judges. A Claude model grading a contest that a Claude model won is fair to question. The rubrics are fixed checklists of 239 weighted items, 79 of them must-pass, which leaves less room for a grader's taste than an open judgment would. It doesn't rule bias out.
• The public tasks may be in the models' training data. That's the price of using tasks anyone can check.
• Twenty-seven tasks is a small sample. One task is worth about 4 points of a model's score, so small gaps are ties.
• This post doesn't split the gain by component. We think structured data, the file system and code execution drive most of it, but we haven't shown here what each one is worth.
• Even a perfect agent wouldn't score 100%. A buy-side investor reviewed all 239 rubric items and found some that are wrong or are judgment calls. We put the achievable score at about 83%, so Opus's 75.6% is 91% of what the rubrics allow.

The method and the full rubric review are in our eval report. Next we're running Gemini 4 Argon and Muse Spark Open in the same setup, and building a finance benchmark with practitioner-reviewed rubrics drawn from how our users work.

Three questions before you pick a model

The model still matters. Opus led in our agent, and Matterfact runs on Claude, with OpenAI, Gemini and fine-tuned models handling the tasks each does best. But in this test the harness moved accuracy by up to 17 points, reordered the models and cut cost per task by up to 15 times. Before you choose, ask:

1. Which harness produced the scores behind the choice, and how close is it to the one we will run?
2. What does a task cost in our own agent, and how long does the slowest one take?
3. Which of our errors would a better prompt fix, and which need a check against the data?