Arcee Trinity Large Benchmarks: What Arcee Actually Published
Four benchmark scores, one comparison model, and the head-to-heads Arcee did not run — read honestly.
Arcee AI published four benchmark scores for Trinity Large Preview: MMLU 87.2, MMLU-Pro 75.2, GPQA-Diamond 63.3, and AIME 2025 24.0. Every number on this page comes from Arcee's own release materials, and nothing has been filled in from estimates.
The comparison model Arcee chose is Llama 4 Maverick. That choice tells you as much as the scores do — it is a US open-weight peer, not one of the Chinese open-weight leaders most buyers are actually weighing.
This page presents the published table, explains what each benchmark measures, and gives an honest read on where Trinity Large sits against DeepSeek, Qwen, Kimi K3, and GLM.
Arcee Trinity Large Benchmark Scores
These are Arcee's published Trinity Large Preview scores, side by side with the comparison model Arcee selected. Higher is better on all four.
| Benchmark | What it measures | Trinity Large Preview | Llama 4 Maverick |
|---|---|---|---|
| MMLU | Broad multi-subject knowledge | 87.2 | 85.5 |
| MMLU-Pro | A harder, reasoning-heavier version of MMLU | 75.2 | 80.5 |
| GPQA-Diamond | Graduate-level science questions | 63.3 | 69.8 |
| AIME 2025 | Competition mathematics | 24.0 | 19.3 |
Trinity Large Preview leads on two of the four and trails on two. It wins on broad knowledge and competition math, and loses on the harder reasoning benchmarks.
That is a mixed result, and Arcee published it as one rather than showing only the two wins. Launch posts do not usually include the losses.
Arcee's published Trinity Large Preview scores against Llama 4 Maverick, the single comparison model Arcee selected. Source: Arcee AI's Trinity Large release post. Verify on Arcee's page before quoting — these change between checkpoints.
Benchmark tables not answering whether Trinity Large can do your actual work? We'll build a task-based evaluation from your real prompts and score the candidates honestly.
Book a ConsultationWhat Each Benchmark Actually Tests
A benchmark score only means something once you know what the test asks. These four cover different capabilities, which is why the results diverge.
None of them measure the thing most business buyers care about: whether the model reliably follows instructions on a long, structured task without drifting.
- MMLU — thousands of multiple-choice questions across 57 subjects. A broad knowledge check, and largely saturated at the top of the market.
- MMLU-Pro — a harder rebuild of MMLU with more answer options and more reasoning steps. Better at separating strong models than MMLU is.
- GPQA-Diamond — graduate-level biology, physics, and chemistry questions written to be resistant to web search. A real reasoning test.
- AIME 2025 — problems from the American Invitational Mathematics Examination. Multi-step math, and a low score here is common for non-reasoning models.
The Thinking Variant Scores Differently
Arcee ships a separate reasoning model, Trinity-Large-Thinking, and reports it on a partly different benchmark set. It produces explicit reasoning traces before answering, so even where the benchmark name matches the Preview table above, the two scores are not directly comparable. Our Trinity Large vs DeepSeek comparison lists the Thinking scores Arcee has published.
Arcee reports that Trinity-Large-Thinking scores 94.7% on τ²-Bench and 91.9% on PinchBench, an agentic benchmark from Kilo. Arcee places it second on PinchBench, behind Claude Opus-4.6.
Arcee frames the PinchBench result as near-frontier capability at a small fraction of frontier pricing. Its own documentation rate card lists the hosted reasoning model at $0.25 per million input tokens and $0.80 per million output tokens, while Arcee's launch post quoted a different output rate — confirm the current figure on Arcee's page, and see our Trinity Large pricing breakdown for the full cost picture.
Treat the two model variants as separate products when you evaluate. A score for the Thinking model tells you nothing about the Preview model, and the reverse is equally true.
How Does Trinity Large Compare to DeepSeek, Qwen, and Kimi K3?
Arcee has not published head-to-head benchmark results against DeepSeek, Qwen, Kimi K3, or GLM. That absence is the most useful fact on this page, and inventing a comparison to fill it would be worse than leaving it open.
The Chinese open-weight leaders publish extensively, and their scores get independently reproduced across public leaderboards. Trinity Large has one vendor-published comparison against one model. The evidence bases are not the same size.
So the honest position is this: on published evidence, Trinity Large is a competitive open-weight model with a thin evaluation record, not a demonstrated leader. It is early. The record will fill in.
| Question | Arcee Trinity Large | Chinese open-weight leaders |
|---|---|---|
| Vendor-published benchmark scores | Yes — four, against one comparison model | Yes — extensive, across many benchmarks |
| Independent third-party verification | Limited so far | Extensive across public leaderboards |
| Head-to-head against the other camp | Not published | Not published against Trinity |
| Benchmarks in common with Trinity's table | Partial — MMLU and GPQA-Diamond are widely reported | Partial |
| Verdict | Judge it on your own task, not the table | Judge it on your own task, not the table |
How to Read a Vendor's Own Benchmark Table
A vendor-published benchmark table is a marketing artifact with real data in it. Both halves of that sentence are true, and the second half is why the numbers are still worth reading.
The tell is the comparison set. Arcee compared Trinity Large to a single model, and a single-comparison table cannot tell you where a model sits in the market.
Our own content-automation routines run across a portfolio of sites, and when we pick a model for that fleet, published leaderboard scores predict almost nothing about the failure we actually care about — instruction-drift on long, structured outputs. A model that tops MMLU can still lose a section heading halfway through a 3,000-word generation. We now evaluate on a fixed set of our own real tasks before a model goes anywhere near production.
Do the same thing at whatever scale you can. Twenty real prompts from your own workflow will tell you more than any published table.
- Check the comparison set — one comparison model is a weak signal, five is a real one.
- Check the checkpoint — Preview, Base, and Thinking variants are different models with different numbers.
- Check whether anyone independent reproduced the score.
- Run your own eval on real tasks from your workflow before committing.
- Match the benchmark to your use case — AIME tells a customer-support team nothing useful.
The Verdict on Trinity Large's Benchmarks
Trinity Large's published scores make it a credible open-weight model, not a benchmark leader. It beats its chosen comparison model on broad knowledge and math, and loses on harder reasoning tests.
The reasoning variant looks stronger on agentic benchmarks, where Arcee reports a second-place PinchBench result behind a frontier hosted model at a small fraction of the price. That is the more interesting number in the whole release.
If you are choosing a model, the benchmark table should not be the deciding input. Trinity Large's actual argument is US origin under a permissive license with weights you can download — and no benchmark measures that.
What you need to run Trinity Large yourself
Trinity Large is a frontier-scale Mixture-of-Experts model, so "running it yourself" is a real infrastructure decision — not something a single laptop or gaming GPU can do. Match the path below to how seriously you need to self-host. For most teams the API or rented GPUs are the right answer; buying hardware only pays off at steady, high volume or when your data can never leave your walls.
| Path | What it is | Best for | Get started |
|---|---|---|---|
| Call the hosted API | Use Trinity Large as a pay-per-token API — zero hardware | Most teams; evaluating before committing | OpenRouter |
| Rent GPUs by the hour | Spin up H100 / A100 nodes on demand, tear them down after | Self-hosting without capital outlay; bursty workloads | RunPod |
| Local on unified memory | A single workstation with enough unified memory to hold a 4-bit quant | One powerful on-prem box; privacy-first solo/SMB use | Apple Mac Studio (M3 Ultra, 512GB) |
| Local on workstation GPUs | Multiple 48GB professional cards for MoE offload / tensor parallelism | Power users and small clusters that want cards they own | NVIDIA RTX 6000 Ada (48GB) |
Once Trinity Large is running, the fastest way to put it to work day to day is inside Cursor — point it at the model through OpenRouter as a custom model. And if you would rather run a model on one affordable box, see Best mini PCs for local AI and Local AI hardware calculator.
Frequently Asked Questions
- Arcee published four scores for Trinity Large Preview: MMLU 87.2, MMLU-Pro 75.2, GPQA-Diamond 63.3, and AIME 2025 24.0. Arcee compared these against Llama 4 Maverick, which scored 85.5, 80.5, 69.8, and 19.3 on the same four tests. Verify current numbers on Arcee's own page, as scores change between checkpoints.
- It depends on the test. On Arcee's published numbers, Trinity Large Preview beats Llama 4 Maverick on MMLU and AIME 2025, and trails it on MMLU-Pro and GPQA-Diamond. That is a split result, which suggests broad knowledge and math strength alongside weaker performance on harder reasoning tasks.
- Arcee has not published head-to-head results against DeepSeek, Qwen, Kimi K3, or GLM, so there is no vendor-published comparison to cite. The Chinese open-weight leaders have far more extensive and independently reproduced benchmark records. On published evidence Trinity Large is competitive with a thin evaluation record, not a demonstrated leader.
- τ²-Bench is an agentic benchmark that tests tool use and multi-turn task completion rather than knowledge recall. Arcee reports Trinity-Large-Thinking scoring 94.7% on it, alongside 91.9% on PinchBench, where Arcee places it second behind Claude Opus-4.6. These scores belong to the Thinking variant, not the Preview model.
- Not extensively, as of publication. The scores come from Arcee's own release materials, which is normal for a recent model launch but is a weaker form of evidence than independently reproduced results. Treat vendor-published numbers as a starting point and run your own evaluation on tasks from your actual workflow.
- Usually none of the published ones. MMLU, GPQA-Diamond, and AIME measure knowledge, science reasoning, and competition math, and few business workflows resemble any of those. A short evaluation built from twenty real prompts out of your own workflow is a far better predictor of production performance.
Test Trinity Large Against Your Own Tasks
Layer3 Labs builds real task-based evaluations for businesses choosing between open-weight and hosted models — so the decision rests on your workload, not a vendor's table. Get a free workflow audit.
Book a Free Workflow Audit