OpenAI GPT-6 Astra: An evaluation

GPT-6 Astra writes less code and quality code than GPT-5.6 Sol, and it concentrates what it still gets wrong into the tier that matters. A stronger model does not pay down verification debt. It moves it.

Every model that writes better code hands your team the same bill: more code to check, and less time to check it. That gap between how fast code arrives and how fast a team can verify it is verification debt.

How was GPT-6 Astra tested for Java coding performance?

Model: GPT-6 Astra, medium reasoning effort

Baseline: GPT-5.6 Sol, medium reasoning effort

Language: Java

Benchmark: 4,444 tasks (HumanEval, MBPP, ComplexCodeEval)

Analyzer: SonarQube algorithmic code analysis. Complexity and code smell densities are per 1,000 lines of code (kLOC). Bug and vulnerability densities, and all category breakdowns, are per million lines (mLOC).

Two terms worth defining before the numbers:

• Cyclomatic complexity: counts independent paths through a function.
• Cognitive complexity: a SonarQube metric that weights nested and deeply branched logic more heavily, reflecting how hard the code is for a human to read.

Neither one tells you whether the code is correct. Both tell you how long review and testing will take.

Two notes on the data. The GPT-5.6 Sol figures used here are the published ones from our GPT-5.6 evaluation, which is the reference for that generation.

Second, the rule mapping behind the category tables is identical for both models on vulnerabilities and code smells.

Key metrics at a glance

How does GPT-6 Astra's pass rate compare to GPT-5?

Astra passes 85.85% of the benchmark. Sol passed 81.99%, and GPT-5.5 before it passed 78.66%. So this generation adds 3.86 points, slightly more than the 3.33 points the previous one added.

Two consecutive releases moving the pass rate by more than three points each is worth noticing. The rate has climbed just over seven points across two generations on the same tasks.

Missing completions dropped to 0.09%, down from 0.25%. Four tasks out of 4,444. The model finishes what it starts.

Roughly one in seven solutions still fails its tests.

How much code does GPT-6 Astra generate?

This is the change that shapes the rest.

Astra generated 656,445 lines across the benchmark. Sol generated 750,198. That is 12.5% less code for the same work, and a higher pass rate on top of it.

It used 65,513 functions to do that, down from Sol's 82,164. So the function count fell faster than the line count, which works out to 10.0 lines per function against Sol's 9.1. Fewer functions, each carrying a bit more.

Comments went the other way, and by a lot. Comment density is 4.2% of output, up from Sol's 1.5%. In absolute terms that is 28,824 comment lines, roughly two and a half times what Sol produced across a larger codebase.

For anyone maintaining this code later, that is a real improvement. Sparse commenting has been a consistent feature of this family, and Astra reverses it.

Does GPT-6 Astra generate more complex code?

Here the two complexity measures disagree, and the disagreement is the point.

Cyclomatic complexity is 220.93 per kLOC against Sol's 220.45. Flat, within a rounding error.

Cognitive complexity is 163.38 per kLOC, up from 143.23. That is a 14% increase.

Cyclomatic complexity counts branches. Cognitive complexity weights nesting and deep branching, the parts that are genuinely hard to hold in your head. So Astra branches about as much as Sol does per thousand lines, but it nests those branches more deeply.

Put that next to the volume result and the picture is consistent. Astra packs the same work into fewer lines and fewer functions. Denser code is the expected outcome, and that is what the cognitive number is showing.

The offset is the commenting. There is more inline context in this code than in anything else we have measured from this family, which helps a reviewer working through denser logic.

What bugs does GPT-6 Astra produce in Java code?

Bug density is 558 per mLOC, down from 724. A 23% reduction.

The whole reduction sits in the bottom two tiers. Major bugs fell 24% and minor bugs fell 37%, and between them that is most of the volume.

Both severe tiers moved the other way. Blocker bugs rose 12%, from 33 to 37 per mLOC. Critical bugs rose 89%, from 37 to 70.

That is the line to plan around. Astra produces fewer bugs per line than Sol, and a larger share of the ones it does produce sit in the two tiers most likely to reach production. A quality gate tuned only to totals will read this as a clean improvement and miss the shift underneath it.

The category breakdown shows where the reduction came from:

Most categories improved, some of them sharply. Null and data value findings fell 64%. Type safety halved. Resource leaks dropped 31%.

Concurrency and threading is still the largest bug category, at 276 per mLOC, but it came down 22% from Sol's 352. That is the first time this category has moved in the right direction in this family, and it is worth calling out because concurrency has led to the bug profile in every model we have evaluated recently.

Exception handling is the one that rose, from 52 to 67 per mLOC.

Concurrency bugs stay expensive regardless of the improvement. They are hard to reproduce, they depend on the environment they run in, and they surface as intermittent failures rather than clean ones. A 22% reduction on the largest category is good news. It does not change where those findings need to be caught, which is testing and analysis rather than review, the kind of algorithmic analysis SonarQube runs automatically on every change

What security vulnerabilities does GPT-6 Astra generate in code?

Vulnerability density is 203 per mLOC, down from 197. A 10% reduction.

Blocker vulnerabilities fell to 3 per mLOC, down from 11. That is a 73% reduction and the lowest blocker figure we have recorded from this family.

Minor findings fell 59% and major findings fell 12%. Critical findings rose 14%, from 125 to 142.

So the security profile does something the bug profile does not. Its most severe tier improved sharply rather than rising, and the critical tier absorbed the difference. Critical findings are now 80% of all vulnerabilities in this output, against 63% for Sol.

The categories show where they sit:

Cryptography misconfiguration is the largest category at 76 per mLOC, up 13% from Sol. This covers weak algorithms, insecure key sizes, and random number generators used in ways they should not be. Insecure system resource handling is second at 61, essentially unchanged.

Those two together account for 137 of the 178, so more than three quarters of the security findings sit in two categories. Both are well covered by automated analysis, which makes this a tractable problem rather than a standing cost on reviewer time.

The clearest improvement is inadequate error handling, which fell from 32 to 5 per mLOC. Hard-coded credentials halved. Path traversal findings do not appear in the Astra output at all.

If you are putting Astra into a pipeline, cryptography configuration is the first place to point automated checks, and it is one SonarQube flag out of the box, exactly as it was for Sol.

How maintainable is GPT-6 Astra-generated code?

Code smell density is 18.47 per kLOC, up from 17.60. A 5% increase, and the one headline measure that moved the harder direction.

Blocker-level smells came down, from 72 to 59 per mLOC.

Most of these barely moved. Collection and generics is up 2%, design and framework up 5%, regex and string formatting up 5%, dead code up 1%. Deprecated API usage fell 47%.

The category that actually moved is cognitive computational complexity, up 65% from 1,213 to 1,999 per mLOC. That is the same finding the cognitive complexity metric reported, showing up a second time through the rules that flag it directly. Denser code produces more of these, and Astra produces denser code.

Collection and generics is still the largest smell category by a wide margin at 7,987 per mLOC. These are raw types where parameterized generics belong, and collection handling that sidesteps type safety. In Java they carry a real cost. They suppress compiler warnings, make refactoring harder, and can hide the kind of bug a properly typed implementation would have caught at compile time.

That category has now led the smell profile in every model we have measured on this benchmark, across both OpenAI and Anthropic.

What are the biggest risks of using GPT-6 Astra generated code in production?

Astra is a stronger generator than Sol on most of what we measure. Higher pass rate, less code, fewer bugs per line, fewer vulnerabilities per line, far better commenting, and fewer blocker findings in both bugs and vulnerabilities. That is a lot moving in the right direction at once.

Three things earn verification attention.

The critical tier is first. Critical bugs more than doubled to 70 per mLOC and critical vulnerabilities rose to 142. Total findings are down, but they are packed more tightly into the band just below the blocker. If your quality gate is tuned to blocker findings alone, it will see an improvement here and miss the shift underneath it.

Density is second. Cognitive complexity per kLOC is up 14%, and the cognitive complexity smell rose 65% alongside it. Each line carries more logic than it did in Sol. Fewer total lines to review does not automatically mean less review time when each one is harder to follow. The heavier commenting helps, and it is worth checking that it holds up in your own code rather than assuming it.

Cryptography configuration is third, and it is the same answer as last generation. At 76 per mLOC it leads the vulnerability profile, and together with insecure system resource handling it accounts for more than three quarters of security findings. Automated analysis covers this well.

The concurrency result deserves a mention on the other side of the ledger. At 276 per mLOC it is still the largest bug category, but it is down 22%, which is the first real movement we have seen there.

Three takeaways:

• Tune your quality gate to the critical tier. Total findings fell, but critical bugs more than doubled and critical vulnerabilities rose 14%. A gate set to blockers alone will read this release as pure improvement and miss the risk that moved one band down.
• Correctness and volume improved together. 85.85% against 81.99%, with 12.5% less code to get there. That combination is uncommon, and it means a smaller review surface as well as a better result.
• Severity concentrated. Bug and vulnerability density both fell, blocker findings fell in both, and the critical tier rose in both. Fewer findings overall, more of them in the band that matters.
• The code is denser. Cognitive complexity is up 14% per thousand lines and the related smell is up 65%. Comment density nearly tripled, which offsets some of that, but review effort does not fall in proportion to the line count.

Astra writes less code, and better code by most measures we track. That does not remove the need to verify what it produces. It moves the question from how much there is to check toward how carefully each piece needs checking.

Here is what to do with this. Do not let a lower total finding count relax your quality gate. Tune it to the critical tier, because that is where this generation moved its defects, and point automated checks at cryptography and concurrency first. That is the work SonarQube is built for: it flags critical-tier bugs, cryptography misconfiguration, and concurrency defects through algorithmic analysis, on every change, at the speed the agent produces them. A stronger model should shorten the path to production, not lengthen the review queue.

Astra now sits near the top of the models we have measured on this benchmark. See exactly where it lands, next to every other generation, on the Sonar LLM Leaderboard.