AI Coding Benchmarks: What They Measure and How Models Score
AI coding benchmarks grade how well a model resolves real bugs, edits code, and completes engineering tasks. This is a reference for what each benchmark tests and how the leading models score — not a ranking of which one to buy.
AI coding benchmarks measure how well a language model can write, edit, and fix code. The strongest ones use real GitHub issues, not toy puzzles. The best-known is SWE-bench Verified, where a model must resolve an actual bug with a working patch.
This page is a data reference. It explains what each coding benchmark tests, its blind spots, and how the top models score on the numbers we publish. It does not rank models or tell you which to pick. For that decision, use our companion guide, Best LLM for Coding.
One rule shapes everything below: treat any single score with care. Many frontier models are partly contaminated on older tests, so testing a model on your own repository beats a headline number.
The Coding Benchmarks That Matter
Seven benchmarks cover most of what teams cite in 2026. Each measures a different slice of coding skill, and each has a real limit. Read the definition and the caveat together — a high score means little without knowing what the test rewards.
- SWE-bench Verified: 500 human-validated real GitHub issues that a model must resolve with a working patch. It is the "Verified" subset of the original SWE-bench. Limit: older repos raise the risk of training contamination, and one pass@1 number hides big harness differences.
- SWE-bench Pro: a harder, contamination-resistant variant built on professional and private repos. We cite 1,865 tasks across 41 repos. Scores drop sharply versus Verified, and that is the point. Limit: still sensitive to the agent harness around the model.
- LiveCodeBench: competitive-programming problems collected after model cutoff dates to avoid contamination. Limit: competitive puzzles are not production engineering, so we de-weight them.
- HumanEval: 164 hand-written Python functions graded by unit tests. Limit: badly saturated, with top models above 90%, and too small to reflect real multi-file work.
- Aider Polyglot: 225 hard exercises across about six languages, testing whether a model edits code in the correct diff format. Limit: it measures edit-format compliance as much as reasoning.
- Terminal-Bench: autonomous terminal tasks in a sandboxed shell. Limit: heavily harness-dependent, so the agent scaffolding can swing scores more than the model.
- Codeforces: a competitive-programming rating (Elo) used as a proxy skill score. Limit: pure algorithm puzzles are a weak proxy for engineering, and Elo is not a percentage.
Benchmarks build the shortlist. We test the top coding models on your real repos and recommend the fit.
Book a ConsultationHow the Models Score
Here are the coding scores we publish, grouped by tier. The scores span several model generations, so do not read them as one clean head-to-head. Note the model version on each figure.
The frontier tier is deliberately light on numbers. Claude Fable 5 and the GPT-5.6 family (Sol, Terra, Luna) carry no published numeric coding scores on our site. Both are gated or preview releases, so we describe them qualitatively rather than invent a figure.
Coding-benchmark scores we publish, colored by tier. Switch tabs to compare SWE-bench Verified, the harder SWE-bench Pro, and Terminal-Bench. Scores span model generations, so note the version on each bar.
- Frontier tier (qualitative only): Claude Fable 5 leads closed-source coding overall but has no published numeric coding score. GPT-5.6 Sol, Terra, and Luna have no published coding scores yet.
- LiveCodeBench and Codeforces: the open-weight Gemma 4 (31B) scores 80.0% on LiveCodeBench and reaches 2150 Elo ("expert level") on Codeforces.
- HumanEval is saturated: both leaders score in the low 90s (see the chart tab), so small gaps here mean little.
- Aider Polyglot: we report this one as a relative delta. Claude Opus 4.5 showed a 10.6% improvement over Sonnet 4.5, with token efficiency 48-76% better.
SWE-bench Verified vs SWE-bench Pro
SWE-bench Pro is the more honest test, and its lower scores are a feature. Verified uses older public repos, so models may have seen similar code in training. Pro uses professional and private repos designed to resist that contamination.
That is why Pro scores fall so far below Verified. A model near 88% on Verified can land near 69% on Pro. The drop is not a failure. It shows how much harder unseen, production-style code is than familiar public issues.
SWE-bench Pro is a stricter, more honest test than SWE-bench Verified. When two models look tied on Verified, their Pro gap often tells you more about real-world work.
How to Read Coding Benchmarks (and Where They Mislead)
A benchmark score is a starting point, not a verdict. Three problems distort almost every coding leaderboard, and knowing them keeps you from over-reading a headline number.
- Contamination: models may have trained on the exact issues in older benchmarks. Testing a model on your own repository is a better signal than a public score.
- Harness sensitivity: the scaffold around the model — how it plans, retries, and edits files — can swing scores more than the model itself. Two teams can report different numbers for the same model.
- Puzzle vs production: competitive-programming benchmarks reward clever algorithms, not the messy multi-file work of real engineering. A puzzle champion can still fumble a routine refactor.
- Tool-call reliability beats benchmark score in an agent loop. A model that scores two points higher but flubs one file edit in ten will waste more of your time.
Benchmarks Give You the Shortlist, Not the Pick
Use benchmarks to narrow the field, then decide on other factors. Scores tell you which models are plausibly strong at coding. They do not tell you which one fits your stack, budget, context window, or agent workflow.
Two models within a few points on SWE-bench can feel very different in daily use. Latency, tool-call reliability, price, and how the model handles your codebase matter more than a one-point benchmark edge.
When you are ready to choose a model for your team, read our companion guide. It turns these scores into a real buying decision.
The Bottom Line on Coding Benchmarks
Coding benchmarks are useful when you read them honestly. SWE-bench Verified and SWE-bench Pro are the strongest signals because they use real issues. HumanEval and pure puzzle tests are the weakest because they are saturated or unrepresentative.
Let the numbers build your shortlist. Then verify on your own repo, weigh tool-call reliability, and check the price. That is how a benchmark reference turns into a decision that holds up in production.
Frequently Asked Questions
- SWE-bench Verified and SWE-bench Pro are the most useful AI coding benchmarks. Both use real GitHub issues that a model must resolve with a working patch, so they reflect production engineering better than puzzle tests. SWE-bench Pro is the stricter of the two because it uses contamination-resistant professional repos. Puzzle benchmarks like HumanEval and Codeforces are weaker signals for real-world work.
- On SWE-bench Verified, top models in 2026 score in the high 80s — Claude Opus 4.8 is cited at about 88.6% and Opus 4.7 at 87.6%. On the harder SWE-bench Pro, scores drop sharply, with leaders in the 60s (Opus 4.8 at 69.2, GLM 5.2 at 62.1). A "good" score depends on the test: expect much lower numbers on Pro than on Verified, and read them as a shortlist, not a verdict.
- Only partly. Coding benchmarks are a starting point, not a verdict. Older benchmarks risk training contamination, scores swing with the agent harness around the model, and puzzle tests do not reflect messy production code. The most reliable signal is testing a model on your own repository. In an agent loop, tool-call reliability often matters more than a one-point benchmark edge.
- SWE-bench Verified is 500 human-validated real GitHub issues from older public repos. SWE-bench Pro is a harder, contamination-resistant variant built on professional and private repos, cited at 1,865 tasks across 41 repos. Scores drop sharply on Pro, which is the point — it is a stricter, more honest test of how a model handles unseen, production-style code.
- Not with numeric scores yet. Claude Fable 5 and the GPT-5.6 family (Sol, Terra, Luna) are gated or preview releases, so we do not publish numeric coding scores for them. Fable 5 is described as leading closed-source coding overall, but that is qualitative. We avoid inventing a figure and lean on the scores of widely available models like Claude Opus 4.8 instead.
- SWE-bench Pro is the best public predictor of real coding performance because it resists contamination and uses professional repos. But no public benchmark beats testing a model on your own codebase. Benchmarks narrow the field; your own repo and your agent workflow decide the winner.
- They test different things. SWE-bench (Verified and Pro) grades whether a model can resolve a real GitHub issue with a working patch inside an existing codebase — the closest proxy to production engineering. LiveCodeBench grades competitive-programming problems collected after a model's training cutoff, so it measures algorithmic problem-solving on fresh puzzles rather than repo-scale bug fixing. Terminal-Bench grades autonomous terminal tasks in a sandboxed shell, so it measures agentic tool-use and command-line reliability more than code-writing skill, and its scores swing heavily with the agent harness. Weight SWE-bench Pro highest for real engineering hires; use LiveCodeBench as a contamination check and Terminal-Bench as a signal for agentic/DevOps work, not a replacement for either.
Not Sure Which Coding Model Fits Your Team?
Benchmarks build the shortlist, but the right pick depends on your stack, budget, and workflow. Layer3 Labs runs your real repositories against the top coding models and recommends the one that fits. Book a free workflow audit.
Book Your Free AI Workflow Audit