Blog
What Independent Benchmarks Say About Opus 5.5
By Tech Nomad · · 8 min read
Two independent benchmark results for Opus 5.5 came out this week, and they don't really agree. One has it in first place. The other has it fast and cheap, but third on secure code once you take out the answers it memorized.
A few days ago I read through Anthropic's Opus 5.5 docs to see what they say got better. That's Anthropic talking about their own model, though. I wanted to see what people who didn't build it found, so I read the first two outside evaluations I could find.
Quick disclaimer: I didn't run any of these tests. All the numbers here come from Artificial Analysis (via heise online) and Endor Labs. The charts are theirs too, credited under each one. The opinions are mine.
Artificial Analysis: first place
Artificial Analysis has an Intelligence Index that combines ten different tests, things like programming and knowledge questions. According to heise's report, Opus 5.5 is now on top with 58 points. OpenAI's GPT-6 Astra and Anthropic's Fable 5.1 got 53 each. All three were run at their highest setting.
Two of those tests are about code. On Terminal-Bench 4.0 (coding agents working in a terminal), Opus 5.5 and Astra are pretty much tied. On SciCode (scientific programming), Opus is 11 points ahead of Astra and 4 ahead of Fable.
One point on Terminal-Bench doesn't mean much. Eleven points on SciCode does, and five points on the whole index is a real lead. Just keep in mind these are all max-effort runs, which matters for the next part.
Cheaper tokens, pricier tasks
Artificial Analysis also worked out what one task from the index costs on average. Opus tokens are cheaper than Astra's, but Astra uses a lot fewer of them, so per task Astra comes out cheaper. Fable 5.1 costs more per token and more per task.
Heise's summary seems fair to me: if you mostly care about performance, try Opus 5.5. If you're watching costs and still want similar results, look at Astra.
One thing I'd add. Opus 5.5 defaults to medium effort, not max, and Anthropic says medium matched or beat Opus 5 on high in their own tests. I haven't seen anyone publish index numbers for Opus 5.5 on medium, so I can't say how that plays out. But the max-effort cost probably isn't what most people will actually pay.
Endor Labs: secure code in real projects
Endor Labstests something much narrower. Their Agent Security League has a coding agent work on real open-source projects, in code that was once part of a security fix. The agent isn't told that. It's only asked to follow security best practices.
Then every patch gets run against two sets of tests. FuncPass means the functional tests pass, so the code works. SecPass means the hidden security tests from the original fix pass too, so the code works and is safe. They ran Opus 5.5 in Claude Code, same setup as for Opus 5 and Fable 5.1, so the model is the main thing that changes.
| Model | FuncPass | SecPass | Memorized |
|---|---|---|---|
| Fable 5.1 | 87.2% | 37.4% | 17 |
| Opus 5.5 | 68.7% | 33.5% | 51 |
| Opus 5 | 73.7% | 32.4% | 38 |
On their full leaderboard, that puts Opus 5.5 third for secure code, behind Claude Code with Fable 5.1 and OpenAI's Codex with GPT-6 Astra (34.6%). For code that just has to work, it's nineteenth. Endor points out that this doesn't line up with Anthropic calling it a new state of the art for coding, at least not on their benchmark.
The number that stuck with me isn't even about Opus 5.5. Fable 5.1, at the top of their board, writes working and secure code on about 37% of these tasks. So whatever model you use, you still need to check the security side yourself.
The memorization thing
This is the part that makes Endor's results worth reading closely. Their tasks come from real, public security fixes, so a model might have seen the answer during training. Endor counts recalling a known fix as cheating, whether it comes from git history, the web or the model's own memory. A pipeline flags suspicious runs, an AI judge reviews each one, and confirmed cases don't count.
For Opus 5.5 they confirmed 51 of those, more than for any other model they've tested. If those counted, it would score 94.4% FuncPass and 52.5% SecPass and be first on security by more than 11 points. Without them it's 68.7% and 33.5%. For comparison, taking out memorized answers cost Fable 5.1 6.7 and 3.9 points.
In one case the model gave itself away by naming the vulnerability's CVE number (its public ID) before it made its first edit. Endor says they added a check for exactly that afterwards.
They also found the memorized tasks were the quick ones: 3.0 minutes on average against 4.4 for the rest. Makes sense. If the model recognizes the task, it doesn't need to read much before writing the fix.
My take: a benchmark built from public data can end up measuring memory as much as skill, and it takes real work to separate the two. Most benchmarks don't publish this kind of breakdown. It's one more reason to try a model on your own code, which it can't have seen.
Where it does well: speed and cost
On speed and cost, Endor found Anthropic's launch claims hold up. Opus 5.5 finished a task in 2.2 minutes at the median and 4.0 on average. Fable 5.1 averaged 9.5 minutes, and Opus 5 averaged 14.1 with 15 timeouts. Opus 5.5 had none.
The cost gap is bigger still. The full Opus 5.5 run cost $116, or $0.33 per task at the median. The same run cost $672 with Fable 5.1 and about $1,116 with Opus 5. According to Endor, about half the gap to Fable comes from the lower price per token and the rest from using fewer of them: about a third of Fable's output tokens, and fewer than half its tool calls (16.6 per task against 42.4).
That lines up with what Anthropic's docs promised, and honestly it's the part I care about most day to day. Fewer steps means less to read, less to check, and less room for the task to drift.
So do they disagree?
Not as much as it looks. They measure different things. Artificial Analysis looks at general ability across ten tests, at max effort, with a model from OpenAI in the mix. Endor looks at one hard, narrow job, secure fixes in real projects, and makes a point of throwing out answers the model memorized.
Where they overlap, they agree: Opus 5.5 is cheaper to run than Fable 5.1 and good at coding. Whether it's the best depends on the task, the effort setting, and whether you count remembered answers.
What I'm taking from it
If speed and cost matter to you, and for most everyday coding they do, Opus 5.5 seems like a reasonable default. Both evaluations back that up.
If you work on security-sensitive code, nothing on Endor's board is close to safe to use without review. Keep your tests, and have someone look at anything that touches logins, user input or permissions.
And if you're choosing between models, a few of your own tasks where you know what good looks like will tell you more than a leaderboard. Leaderboards are a fine place to start, just not to stop.
Sources
- External benchmarks: Claude Opus 5.5 overtakes OpenAI's Astra and Fable 5.1, heise online, September 25, 2026. Reports the Artificial Analysis results; both Artificial Analysis charts above are from this article.
- Artificial Analysis, publisher of the Intelligence Index.
- Opus 5.5: 6x cheaper and 2x faster than Fable 5.1, but only 33.5% of code is secure, Endor Labs, September 24, 2026. Source of the Agent Security League results and the task-duration chart.