OpenAI GPT-5.6 Sol and Terra: An evaluation

Every model that writes better code hands your team the same bill: more code to check, and less time to check it. That gap between how fast code arrives and how fast a team can verify it is verification debt, and a stronger model does not pay it down. It moves it.

GPT-5.6 is a better code generator than the model before it. This evaluation is about what that improvement actually costs you, and where.

OpenAI's GPT-5.6 arrived in three variants: Sol, Terra & Luna. We evaluated Sol and Terra here; Luna is out of scope for this run. Same generation, three different builds, and the benchmark data shows they do not behave the same way. 

We ran the eval for both Sol and Terra through Sonar's LLM evaluation framework, which measures generated code against the same analysis  we'd apply to anything a developer wrote.  Same 4,444 Java tasks, same quality profile, same reasoning effort. Because we re-ran GPT-5.5 through the same analysis, every comparison below is apples to apples.

The short version: Sol writes more correct code than GPT-5.5, and it is easier to read. But the security risk moved, and it moved into two places a code review will not reliably catch. Let’s get into the details and look at the full data.

How was GPT-5.6 Sol and Terra evaluated?

Models: GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.5 (baseline)

Language: Java

Benchmark: 4,444 tasks (HumanEval, MBPP, ComplexCodeEval)

Reasoning effort: medium, for all three

Analyzer: SonarQube algorithmic code analysis. Complexity and code smell densities are per 1,000 lines of code (kLOC); bug and vulnerability densities, and all category breakdowns, are per million lines (mLOC).

Two terms worth defining before the numbers:

• Cyclomatic complexity: counts independent paths through a function.
• Cognitive complexity: a SonarQube metric that weights nested and deeply branched logic more heavily, reflecting how hard the code is for a human to read.

Neither one tells you whether the code is correct. Both tell you how long review and testing will take.

In the severity tables below, the per-severity rows are counted against total issues raised, so they sum slightly above the headline density figures for the 5.6 variants. Read the severities as a distribution, not as a total.

Key metrics at a glance

Functional skill

Sol passes 81.99% of the benchmark, up from 78.66% for GPT-5.5. That's a 3.3 point gain on identical tasks.

It also leaves roughly one in five solutions failing its tests. You can't tell in advance which one. That's not a knock on the model, it's just the arithmetic of the thing.

Missing completions came in at 0.25% for Sol. Low, and close enough to GPT-5.5's 0.27% that the model finishes what it starts.

How much code does GPT-5.6 Sol generate compared to GPT-5.5?

Sol generated 750,198 lines across the benchmark, about 6.8% more than GPT-5.5. But it did that across 82,164 functions, down from 92,206. More code, fewer functions.

Work that out per function and you get 9.1 lines, up from 7.6. Sol writes longer functions that do more. Not a problem on its own, though it does mean each unit you review carries more logic than it used to.

Comments dropped to 1.5% of output, from 2.0%. In absolute terms that's 11,713 comment lines across 750,000 lines of code. For every 100 lines a reviewer opens, roughly one and a half carry any explanation. Well-named functions and variables can carry that load instead, and often do. Just worth knowing going in.

Does GPT-5.6 Sol generate more complex code than GPT-5.5?

Here's a nice surprise: complexity densities came down.

Cyclomatic complexity is 220.45 per kLOC, down from 250.57. Cognitive complexity is 143.23 per kLOC, down from 151.27. Sol writes more total code, but each thousand lines of it branches less than GPT-5.5's did.

That matters because cognitive complexity tracks how hard code is for a person to hold in their head. Nested conditionals, loops inside loops, branching where you have to track several states at once. Lower is easier to review, easier to test, easier to change without breaking something. Sol moved in the right direction on both counts.

Bug density and severity

Bug density is 724 per mLOC, up from 504. That's an increase of about 44%.

Blockers and criticals are the two that cause production incidents, and both moved up, to 45 and 33 per mLOC. The bigger movement is in the long tail. Minor bugs went from 205 to 351 per mLOC, a 71% jump. Those don't break anything today. They accumulate, slow down future changes, and occasionally surface as real bugs once the codebase shifts around them.

The largest categories are where this gets specific:

Concurrency and threading more than doubled, and it now sits well clear of everything else at 349 per mLOC. Concurrency bugs are the expensive kind. They're hard to reproduce, they depend on the environment they run in, and they show up as intermittent failures that eat days of debugging time. 

Resource leaks, exception handling, and type safety all improved slightly. Small moves, but they're moves in the right direction.

What security vulnerabilities does GPT-5.6 Sol generate in code?

Vulnerability density is 197 per mLOC, up from 68. This is the biggest change in the evaluation, and the severity split is the part worth reading closely.

Blocker vulnerabilities halved, from 18 to 9 per mLOC. Everything else went up, with the critical category carrying most of it.

So the shape of the profile changed, not just the volume. In GPT-5.5, blockers made up 27% of all vulnerabilities. In Sol they're under 5%, and criticals are just under 62%. Fewer of the most severe findings, considerably more of the tier below.

The category breakdown explains where the increase came from. This is the full picture for both variants, and the rows add up to the density, so nothing is hiding:

Injection attack findings don't appear in the GPT-5.5 output, and together with the insecure in insecure system resource handling they account for 76 of Sol's 129-point increase.

Cryptography misconfiguration is the other big mover, from 17 to 67 per mLOC, and it's now the largest single category. This covers weak algorithms, insecure key sizes, and random number generators used in ways they shouldn't be. It's also a category static analysis handles reliably, which is the useful part.

Some categories went the other way. Path traversal and injection dropped from 10 to 4 per mLOC, and XXE from 9 to 1. Both were already low and both got lower.

If you're wiring Sol into a pipeline, cryptography and system resource handling are where review time pays for itself. Those two are most of the story.

How maintainable is GPT-5.6 Sol generated code?

Code smell density is 17.60 per kLOC, against 17.05 for GPT-5.5. That's a 3.2% difference. Effectively flat.

Which is a genuinely good result given the extra volume. Blocker-level smells even came down, from 78 to 72 per mLOC.

The drivers are the same ones we've seen from this family:

Collection and generics issues are the largest named category, at 7,803 per mLOC. Raw types where parameterized generics belong, collection handling that sidesteps type safety. In Java these have a real cost: they suppress compiler warnings, make refactoring harder, and hide the kind of bug a properly typed implementation would have caught at compile time.

Dead and unused code dropped by about 40%, from 1,434 to 862 per mLOC. Deprecated API usage went the other way, from 189 to 321.

How does GPT-5.6 Terra differ from GPT-5.6 Sol?

Terra takes a visibly different approach to the same 4,444 tasks, and if you're choosing between the two variants the differences are worth having in front of you.

Start with volume. Terra wrote 617,132 lines, 12.2% fewer than GPT-5.5 and 18% fewer than Sol. It used 70,378 functions to do it, roughly a quarter fewer than GPT-5.5. Terra is the concise one.

Its pass rate is 79.96%, a little above GPT-5.5 and a little under Sol. It also had the fewest missing completions of the three, at 0.18%.

The tradeoff shows up in density. Because Terra packs the same work into fewer lines, its per-kLOC figures run higher. Code smell density is 23.31 per kLOC, 37% above GPT-5.5. Cognitive complexity is 161.53 per kLOC, the highest of the three. Comments are 0.9% of output, which works out to 5,600 comment lines in total.

Most of the extra smell volume is minor severity, which went from 6,145 to 10,367 per mLOC. Blocker-level smells actually came down, from 78 to 62 per mLOC, the best of the three. And collection and generics issues nearly doubled to 11,843 per mLOC, which is the single largest driver.

On security, Terra lands close to Sol: 203 vulnerabilities per mLOC, with cryptography misconfiguration at 76 and blockers at 11. Its bug profile is close too. Bug density is 763 per mLOC, about 51% above GPT-5.5, and concurrency and threading leads at 350 per mLOC.

The short version: Sol gives you the higher pass rate and flat maintainability at slightly more volume. Terra gives you meaningfully less code and the lowest blocker-smell density, with more findings per line. Neither is strictly ahead of the other, and which one fits depends on whether your bottleneck is review volume or review depth.

How do GPT-5.6 Sol and Terra token usage compare to GPT-5.5?

One more thing from the data, with a caveat attached.

Input tokens were effectively identical across all three runs at about 1.32 million, which is what you'd expect from the same benchmark. Output tokens weren't. GPT-5.5 produced 5.55 million, Sol produced 9.58 million, and Terra produced 8.37 million. That's 73% and 51% more output for code volume that moved a lot less.

Both 5.6 variants also report reasoning tokens: 4.00 million for Sol, 3.66 million for Terra.

What are the biggest risks of using GPT-5.6 generated code in production?

GPT-5.6 writes more correct code, and code that is easier to read. Neither of those removes the need to verify what it produces. They change where verification has to point, and this evaluation says exactly where.

Two areas earn that attention, and both are places a read-through tends to miss.

Concurrency is the first. At 352 per mLOC for Sol and 350 for Terra, threading is comfortably the largest bug category in both variants. Concurrency bugs are intermittent, environment-dependent, and expensive to reproduce. If what you generate is concurrent by nature, plan to catch these at the testing and analysis stage, because they do not reliably surface in review. 

Security configuration is the second, and it is narrower than you might expect. Cryptography misconfiguration and insecure system resource handling are the top two vulnerability categories in both variants, and between them they explain most of the rise over GPT-5.5. Both are exactly the kind of finding automated analysis that catches reliably that makes this a solvable problem, not a standing tax on reviewer time.

The variant choice is a real one. Sol for the higher pass rate and flat maintainability. Terra if less code matters more to you than fewer findings per line.

Three takeaways:

• Correctness improved, clearly. Sol's 81.99% pass rate is 3.3 points above GPT-5.5 on identical tasks, and complexity density came down at the same time.
• The security profile changed shape. Blocker vulnerabilities halved to 9 per mLOC while criticals moved to 125. Cryptography and system resource handling are where verification should focus.
• The two variants are genuinely different. Terra writes 12% less code than GPT-5.5 with the lowest blocker-smell density of the three, at a higher findings-per-line rate. Pick the profile that matches your bottleneck.

Full evaluation results for both GPT-5.6 variants, along with every other model we've measured, are on the Sonar LLM Leaderboard.