Tasks passed
Percentage of tasks where the model succeeded on at least one of three attempts (also called Pass@1). Shown with a 95% confidence interval when available.
How it works
Methodology
Finance Benchmark grades models on verifiable finance tasks. The two numbers you will see most often are Tasks passed (share of tasks solved at least once) and Finance Index (equal-weight domain composite). Details below.
Key metrics
Finance Index
Equal-weight average of per-category scores (0–100), excluding saturated categories (e.g. knowledge and analysis) that all frontier models ace.
Consistency
Average number of successful runs per task across the three attempts (for example 2.1/3).
Held-out tasks
Three tasks per category stay private to reduce contamination. Public catalog scores and leaderboard scores use the full private set during harness runs.
Design principles
The v2 set covers 73 tasks across eight finance domains. Scores use three attempts per task at temperature 0; overlapping confidence intervals are statistical ties. From harness 0.2.0, runs can also record calibration and failure taxonomy metadata.
Verifiable ground truth
Every task has an objective answer: MCQ letter match, numeric tolerance, or executed code compared to reference implementations.
Contamination resistance
Quant and v2 hard tasks use seeded parameters so answers are reproducible but not easily memorized. Three tasks per category are held out from the public catalog.
Reproducibility
Temperature 0, three runs per task, versioned harness and task sets, pinned prompts. Full per-attempt responses are stored in result JSON.
Headroom
Includes Greeks precision, multi-step pricing, and industry workflows where frontier models still underperform conceptual finance questions.
Held-out evaluation set
The public task catalog shows a subset of tasks. Three tasks per category are withheld to reduce benchmark contamination. Leaderboard scores are computed on the full private set during harness runs. See the task catalog for what is published.
Scoring rules
| Category | Metric | Tolerance |
|---|---|---|
| Knowledge | Letter accuracy | Exact match |
| Analysis | Numeric accuracy | 1% relative |
| Quant | Code execution vs reference | 0.1% (0.1–1% for MC) |
| v2 hard domains | Numeric or code vs gold reference | Per-task tolerance |
How to reproduce
pip install -e . finbench run --model anthropic/claude-sonnet-4-6 --tasks all --runs 3 finbench publish results/<model>_<timestamp>.json
Want your model on the board? See Collaborate.