Claude Opus 5: An evaluation review & metrics benchmarks

Claude Opus 5 is Anthropic's latest flagship, released on July 24. Anthropic describes it as delivering frontier intelligence at half the price of Claude Fable 5, and calls out one capability in particular: Opus 5 is unusually good at verifying its own work and catching logical flaws while it plans.

That's an interesting claim to test with algorithmic analysis, so we ran it through Sonar's LLM evaluation framework against the same Java benchmark we use for every model.

The short version: correctness improved a lot. Bug density, vulnerability density, and cognitive complexity per line fell, while code smell density, overall issue density, and cyclomatic complexity rose. There's also a volume story underneath it that changes what verification looks like in practice.

What was measured

Model: Claude Opus 5 Thinking (adaptive thinking mode)

Baseline: Claude Opus 4.8 Thinking

Language: Java

Benchmark: 4,441 tasks (158 HumanEval, 386 MBPP, and 3,897 ComplexCodeEval tasks)

Analyzer: SonarQube systematic code analysis. Density metrics are per 1,000 lines of code (kLOC); category breakdowns are per million lines (mLOC).

Two terms worth defining first:

• Cyclomatic complexity: counts independent paths through a function.
• Cognitive complexity: a SonarQube metric that weights nested and deeply branched logic more heavily, reflecting how hard the code is for a human to read.

Neither one tells you whether the code works. Both help indicate how much logic reviewers and testers will need to reason through.

Opus 4.8 is the baseline here, since it's the release Opus 5 directly follows. If you read our Opus 4.7 write-up, the refreshed leaderboard now reports 82.72% for 4.6, 83.66% for 4.7, and 82.9% for 4.8. Those three releases remained within one percentage point of each other.

The figures below use the leaderboard's August 14, 2026 refresh. The bug, vulnerability, and code smell category mappings are identical for the two models, so the category comparisons are direct. Headline metrics use the leaderboard's bug, vulnerability, and code smell measures, while the severity tables use reliability, security, and maintainability impact measures, so the two sets of measures are not expected to reconcile exactly.

Key metrics at a glance

What is Claude Opus 5's functional pass rate?

88.6%. That's the number that makes this evaluation interesting.

Across the 544 HumanEval and MBPP tasks with executable tests, Opus 4.8 achieved an 82.9% pass result. Opus 5 achieved 88.6%, a gain of 5.7 percentage points on the same tasks. ComplexCodeEval contributes to the code analysis but not to the functional pass rate. To put that in context, the three Opus releases before this one all landed between 82.72% and 83.66%. This is a step, not a drift.

Missing completions also improved, from 1.22% to 0.83%. The model finishes what it starts more reliably than its predecessor did.

Roughly one in nine solutions in the 544-task test-backed subset still doesn't pass its tests. Better odds than the model before it, and still a ratio that needs something downstream to catch it.

Does Claude Opus 5 generate more code than Opus 4.8?

Here's the change that shapes everything else. Opus 5 generated 916,813 lines of code across the benchmark. Opus 4.8 generated 391,456. That's 2.3 times more code for the same 4,441 tasks.

It also used 122,650 functions, up from 48,734. So the function count grew faster than the line count. More code, more functions, each one slightly tighter. That pattern points toward more decomposition rather than simple sprawl.

Comment lines went from 6.9% to 10.5% of code plus comment lines. In absolute terms that's 107,517 comment lines, up from 29,024. For scale, the two GPT-5.6 variants we evaluated this month came in at 1.5% and 0.9%. Opus 5's comment-line density is 7 to 12 times higher than those results.

For anyone maintaining this code later, that provides more inline context and more signposts, as long as the comments remain accurate.

Complexity

Cognitive complexity is 132.29 per kLOC, down from 146.40. That's a 10% reduction.

Cyclomatic complexity moved slightly the other way, from 228.40 to 237.82 per kLOC, a 4% increase.

Neither complexity measure moved much once normalized. Cognitive complexity is down 10%, cyclomatic up 4%. Given 2.3× more code and 2.5× more functions, shifts of that size say less about how the model writes than about what it wrote this time. Complexity density is roughly flat.

Bug density and severity improvements in Claude Opus 5

Bug density is 576 per mLOC, down from 672. A 14% reduction, and it happened while output volume more than doubled.

All four severity levels improved. BLOCKER reliability impact issues fell from 38%, from 66 to 41 per mLOC. High-severity findings came down from 33 to 28. Medium from 286 to 241, and Low from 291 to 265.

That’s the cleanest reliability result. BLOCKER is the highest severity tier, and it moved in the right direction alongside every tier below it.

The category breakdown shows where the overall reduction came from:

Nine of the 11 rows improved. Exception handling dropped 52%, from 66 to 32 per mLOC. API contract violations more than halved. Control flow mistakes fell 51%. Resource leaks and type safety issues also came down.

Concurrency and threading nearly doubled, from 105 to 205 per mLOC, and became the largest bug category by a clear margin. Null and data-value findings also increased, from 56 to 91 per mLOC.

That's a familiar pattern. Concurrency was also the top bug category in all three GPT models we evaluated this month, and the reasons are inherent. Threading bugs are hard to reproduce, they depend on the environment they run in, and they show up as intermittent failures rather than clean ones. They are easy to miss in code review, which makes testing and analysis especially important.

What security vulnerabilities does Claude Opus 5 generated code contain?

Vulnerability density is 251 per mLOC, down from 314. A 20% reduction.

The security-impact severity profile is where this gets notable:

BLOCKER security-impact issues went from 77 to 19 per mLOC. That's a 75% reduction. MEDIUM-severity findings dropped 86%.

HIGH severity findings rose, up 29%. BLOCKER, MEDIUM and LOW also came down, so three of the four tiers improved and the most severe band thinned out considerably.

The category breakdown uses the same rule mapping for both models:

XML external entity findings fell from 23 to 4 per mLOC, hard-coded credentials dropped from 59 to 14, and injection findings declined from 18 to 7. Path traversal and injection findings fell from 20 to zero in this snapshot.

Cryptography misconfiguration moved in the other direction, from 43 to 103 per mLOC, and is now the largest vulnerability category.

Either way, it's the category to point automated checks at first. Crypto misconfiguration includes weak algorithms, insecure key sizes, and random number generators used in unsafe ways, which automated analysis can surface before review.

How maintainable is Claude Opus 5 generated code?

Code smell density is 19.69 per kLOC, up from 16.76. A 17% increase, and one of the metrics in this evaluation that moved in a worse direction.

MEDIUM-severity maintainability-impact issues came down 9%. LOW-severity findings went up 48%, and that's where most of the increase lives.

The category view shows this isn't a broad shift. It's concentrated:

Four rows went up. Six came down, four of them by 40% or more.

Collection and generics issues more than doubled, from 3,929 to 8,901 per mLOC, and it's now the dominant smell category by a wide margin. These are raw types where parameterized generics belong, and collection handling that sidesteps type safety. In Java they carry a real cost: they suppress compiler warnings, make refactoring harder, and can hide the kind of bug a properly typed implementation would have caught at compile time.

Naming and documentation smells increased 73%, which is worth reading alongside the comment density result. More annotation overall, more findings on how some of it is written.

Most other named categories moved in the other direction. Design and framework issues fell 31%. Dead code fell 45%. The cognitive complexity smell dropped 45%, which lines up neatly with the cognitive complexity metric coming down.

How does Claude Opus 5 code volume affect total findings

The density results are mixed. Bug density fell 14%, vulnerability density fell 20%, and cognitive complexity fell 10%. Code smell density rose 17%, overall issue density rose 16%, and cyclomatic complexity rose 4%. Comment-line density increased from 6.9% to 10.5%.

Absolute counts are a different picture, because there's 2.3 times more code.

Bugs roughly doubled, and vulnerabilities increased 1.9 times. Code smells rose 2.8 times, and since smells are about 96% of all findings, total issues came in at 2.7 times the 4.8 figure. The increase reflects 2.3 times more code together with higher code smell and overall issue density, even though bug and vulnerability density fell.

Both readings are true, and the gap between them is entirely the 2.3× volume multiplier. Density asks how clean the code is per unit written; absolute counts ask how much work exists. Neither answers the other's question, and substituting one for the other produces a specific error. Cite the density improvement to argue review load is unchanged and you'll under-resource by roughly a factor of three. Cite the absolute counts to argue the model writes worse code and you'll be wrong about bugs and vulnerabilities, which improved on a per-line basis.

The caveat sits underneath both: density is a fair comparison only if the volume itself is neutral. If Opus 5 writes 2.3× more code for the same task, that verbosity is a quality signal in its own right, and dividing by it flatters the result. Density is the right measure for comparing models on equivalent output. It's the wrong one for deciding whether the extra output was warranted.

A note on tokens

Input tokens were effectively identical across both runs at about 2.07 million, which is what you'd expect from the same benchmark. Output was not. Opus 4.8 produced 6.00 million output tokens; Opus 5 produced 21.71 million. That's 3.6 times more output for 2.3 times more code.

What does this mean for teams using Opus 5?

The correctness gain is the headline and it's a real one. An 88.6% pass result across the 544 HumanEval and MBPP tasks with executable tests, after three releases that all landed between 82.72% and 83.66%, is a genuine step forward. And it arrived alongside lower bug density, lower vulnerability density, lower cognitive complexity, and better commenting. Models don't usually improve on that many axes at once.

Anthropic's own framing, that Opus 5 is strong at verifying its own work, is consistent with what we see. Exception handling findings dropped 52%. BLOCKER security-impact issues dropped 75%. Control flow mistakes fell 51%. Every reliability severity tier came down.

Three places to point verification effort.

Concurrency is first. At 205 per mLOC it's the largest bug category and it nearly doubled from 4.8. If what you're generating is concurrent by nature, prioritize testing and analysis for threading issues alongside code review.

Cryptography configuration is second. At 103 per mLOC it's the largest vulnerability category. It's also well-covered by automated analysis, so this is a solvable problem rather than a standing tax on reviewer time.

Collections and generics is third. At 8,901 per mLOC it dominates the smell profile, and it's the kind of issue that's cheap to fix at generation time and expensive to unpick after a codebase has grown around it.

And then there's volume. Opus 5 writes 2.3 times more code than the model it replaces, with more comments and smaller functions. Bug density, vulnerability density, and cognitive complexity improved per line, while code smell density, overall issue density, and cyclomatic complexity worsened. That's the tradeoff to plan for, and it's a manageable one when the verification layer is automated rather than manual.

Three takeaways:

• Correctness took a real step. 88.6% versus 82.9% across the same 544 HumanEval and MBPP tasks with executable tests, after three releases that stayed within one percentage point of each other.
• The severity results include a strong security gain. BLOCKER security-impact issues dropped 75% to 19 per mLOC, every reliability severity came down., BLOCKER included, from 66 to 41 per mLOC. Bug and vulnerability density are both down.
• Volume is the thing to plan for. 2.3 times more code means 2.7 times the total findings, even with lower bug and vulnerability density. Verification effort rises with code volume as well as functional correctness.

Opus 5 achieved a higher functional pass rate and lower bug density, vulnerability density, and cognitive complexity than the model it replaces, while producing more code with higher code smell density, overall issue density, and cyclomatic complexity. That doesn't remove the need for verification. It changes where verification should point, and it makes the case for pointing it automatically.

Full evaluation results for Claude Opus 5, along with every other model we've measured, are on the Sonar LLM Leaderboard.