These are the publicly available benchmarks the market uses to evaluate AI. They are a starting point. The best evaluations are built on custom datasets grounded in specific workflows. That is what Caliper Lab specialises in.
Whether a model can answer specific questions from SEC filings using the source document as context.
Open-book QA on real 10-K, 10-Q, and 8-K filings. Designed as a minimum performance standard for financial LLMs. Built by PatronusAI with annotated evidence strings.
AccessLoads directly. 150 rows.
View datasetWhether a model can work from annual reports and balance sheets to derive ratios and performance metrics rather than quote a stated figure.
QA pairs drawn from annual reports and financial statements, covering asset turnover, ratios and performance metrics. Structured into query, answer and context passage fields.
AccessGated. Accept the terms on HuggingFace before the files will download.
View datasetWhether a model can locate and interpret financial disclosures across the narrative sections of 10-K filings.
QA pairs grounded specifically in 10-K SEC filings with supporting context passages. Covers MD&A, risk factors, and business descriptions.
AccessLoads directly. 7,000 rows.
View datasetWhether a model can combine data from tables and prose in the same financial report to answer multi-step numerical questions.
Hybrid QA over real 10-K and 10-Q filings across 2,757 documents. Requires numerical reasoning across tables and text. Human expert baseline: 90.8% F1.
AccessRaw JSON files, no parquet conversion. Download the tatqa_dataset_*.json files directly.
View datasetWhether a model can handle scanned or image-based financial documents, the format most real-world legacy documents arrive in.
Document visual QA extension of TAT-QA across 2,758 documents. Questions over page images of financial reports. Relevant for vendors processing scanned or low-quality documents.
AccessRaw JSON files, no parquet conversion. Download the tatdqa_dataset_*.json files directly.
View datasetWhether a model can carry out multi-step numerical reasoning across a financial document where the evidence sits in the page image, not the text layer.
Multimodal reasoning over financial documents, built for the complex multi-step tasks an analyst actually performs rather than single-fact lookup.
AccessImages and text in separate folders plus test.json. No table viewer, download the folders.
View datasetAgent benchmarks, where the test is whether a system can find the right number in the right place, not whether it can reason over a passage handed to it.
Whether an agent retrieves specific numeric values from a structured financial database rather than guessing from web search results.
Daloopa's agent retrieval benchmark across 500 questions and 14 agent configurations spanning Anthropic, OpenAI and Google. Headline finding: Claude Opus scored 90.8% with structured APIs against 19.8% with web search.
AccessLoads directly. Includes questions, responses, scores and the full tool_traces parquet.
View datasetWhether an agent picks the right document type first and then locates the key passage, separating document selection from passage retrieval.
Agentic retrieval benchmark for financial QA on S&P 500 firms, expert-annotated. The two-stage split is the useful part: most retrieval failures are document selection failures. ACM ICAIF 2025.
AccessJSON question files plus a datasheet. No table viewer, download benchmark_questions.json.
View datasetWhether an agent can execute a multi-file, multi-tool, multi-step financial workflow end to end, which academic single-turn benchmarks do not test.
Chinese-language benchmark for AI agents in the financial domain. Relevant as the agentic evaluation frontier moves toward multi-step execution flows.
AccessChinese-language. Public questions ship as a single .xlsx plus an attachments folder.
View datasetA single reference point for the full financial AI evaluation landscape across extraction, QA, forecasting, risk management and decision-making.
FINOS Foundation leaderboard covering 42 financial datasets across 7 categories. The best single map of what already exists before building anything new.
AccessA live leaderboard, not a download. Use it to find the underlying datasets.
View datasetWhether a model correctly classifies sentiment in financial news text, the baseline for any research or monitoring product making tone or signal claims.
Sentiment classification for financial news sentences. Annotated by 16 people with finance backgrounds across three classes. The most widely used financial NLP baseline.
AccessUses a legacy loading script. Data files are present, but the HuggingFace viewer cannot render it.
View datasetWhether a system holds up on news at volume, for monitoring, signal extraction and language modelling rather than document QA.
Unified corpus of financial news spanning 1990 to 2025 across 24 subsets. Built for stock trading reinforcement learning, language modelling and NLP research.
AccessGated. Accept the terms on HuggingFace. Large, so take a sample subset.
View datasetWhether a small fine-tuned model can match a frontier model on financial sentiment, which decides whether sentiment is worth paying frontier prices for.
The instruction-tuning set behind FinGPT, covering financial news and social media. LoRA fine-tunes on this data reportedly outperform GPT-4 on financial sentiment.
AccessLoads directly. 76,772 rows.
View datasetWhether a model can identify and extract specific clause types from commercial contracts, the core task in M&A diligence and contract review.
Contract Understanding Atticus Dataset. 13,000+ expert annotations across 41 clause types, built over a year by legal experts. Conservative cost to replicate: $2M.
AccessLoads directly. 84,325 rows.
View datasetWhether a model can reason across a wide range of legal task types, from contract analysis to statutory interpretation.
162 legal reasoning tasks gathered from 40 contributors including lawyers, law professors and practitioners. Stanford / HazyResearch.
AccessLoads directly. 91,750 rows across all tasks.
View datasetWhether a model can classify whether a contract supports, contradicts or is silent on a given hypothesis, the core NDA and compliance review task.
Natural language inference over contracts. Three-way classification across NDA text. Used as a foundational test in both LegalBench and LegalBench-RAG.
AccessLoads directly. 20,107 rows.
View datasetWhether the retrieval layer finds the right clause before the model answers, which isolates retrieval quality from generation quality.
Retrieval benchmark for RAG over legal contracts, human-annotated by legal experts across a 79M-character corpus of 714 documents drawn from CUAD, MAUD, ContractNLI and PrivacyQA. A mini version exists for fast iteration.
AccessGitHub. The README links the corpus download; the repo itself holds the generation code.
View datasetWhether a model flags real legal risk in a clause, and whether it avoids the work by answering no related clause when there is one.
Clause-level legal risk identification over 41 risk categories, measuring correctness, output effectiveness and laziness. The laziness metric is the unusual one and it is the failure mode buyers notice.
AccessA methodology and results paper, not a separate corpus. It runs on CUAD, listed above.
View datasetThese are the benchmarks frontier models are evaluated against. Useful as a general capability baseline. Do not mistake high scores here for task-specific performance on professional workflows.
Broad academic knowledge across 57 subjects including finance, law, economics and business. The standard measure of frontier model breadth.
Measuring Massive Multitask Language Understanding. Most frontier models now score above 85%, so it is a general baseline rather than a workflow performance measure.
AccessLoads directly. 231,400 rows.
View datasetTask types designed to exceed model capability at release. Useful for identifying categories where frontier models still fail.
Beyond the Imitation Game. 200+ challenging tasks, useful for finding the failure frontier rather than measuring average performance.
AccessScript only, no data files in the repo. Downloading the Files tab gets you bigbench.py, not the data.
View datasetEvaluation design itself: accuracy, calibration, robustness, fairness and efficiency measured together rather than a single headline score.
Holistic Evaluation of Language Models from Stanford CRFM. The reference methodology for multi-dimension evaluation. Worth reading even if you never run it.
AccessA runnable framework, not a download. Scenarios pull their own data.
View datasetWhether an agent can analyse spreadsheets and business reports across question answering, chart generation and file output.
AI Data Analytics Benchmark, spanning spreadsheets, financial statements and business reports with separate evaluators for answers, visualisations and generated files. Best model scores 59.4% pass-at-1, so it is far from saturated.
AccessGitHub. Tasks and evaluator code live in the repo.
View datasetConsulting, advisory, and complex multi-document workflows have no meaningful public benchmarks. The best evaluations are built on custom datasets grounded in real tasks. Caliper Lab designs and builds evaluation datasets for AI teams that need to measure what actually matters in their domain.