OpenAI GPT-6 Sol: An evaluation

Compared with GPT-6 Astra, GPT-6 Sol writes 27% less code and produces 22% fewer findings in total, at a pass rate 2.76 points lower and with more findings per line of code.

GPT-6 Astra was the release where correctness and volume improved together. Astra has the higher pass rate and lower issue density, while Sol generates substantially less code and fewer findings overall..

We ran Sol through Sonar's LLM evaluation framework against the same Java benchmark we use for every model, and against the same Astra run we published earlier this month.

How was GPT-6 Sol evaluated?

Model: GPT-6 Sol, medium reasoning effort

Baseline: GPT-6 Astra, medium reasoning effort

Language: Java

Benchmark: the same Java benchmark we use for every model, covering HumanEval, MBPP and ComplexCodeEval. Both runs record 4,444 tasks. All of them are analyzed for code quality and security, which is where every density figure below comes from. 544 of them ship with executable tests, and the pass rate covers only those.

Analyzer: SonarQube systematic code analysis. Complexity and code smell densities are per 1,000 lines of code (kLOC). Bug and vulnerability densities, and all category breakdowns, are per million lines (mLOC).

Two terms worth defining first:

• Cyclomatic complexity: counts independent paths through a function.
• Cognitive complexity: a SonarQube metric that weights nested and deeply branched logic more heavily, reflecting how hard the code is for a human to read.

Neither one tells you whether the code works. Both help indicate how much logic reviewers and testers will need to reason through.

How do GPT-6 Sol and GPT-6 Astra compare on key metrics?

What is GPT-6 Sol's functional pass rate?

83.09% across the 544 test-backed tasks, against 85.85% for Astra. That is 2.76 points lower.

For context, GPT-5.6 Sol passed 81.99% and GPT-5.5 before it passed 78.66%. So GPT-6 Sol still sits above the generation before it, just below its sibling.

Missing completions are identical in both runs at 0.09%, which is 4 tasks out of 4,444 where no usable code came back. Both variants finish what they start.

Roughly one in six solutions in the test-backed subset fails its tests, against one in seven for Astra. That is the clearest difference between the two variants, and it is the number to weigh against everything Sol does well below.

How much code does GPT-6 Sol generate compared to Astra?

Sol generated 480,125 lines across the benchmark. Astra generated 656,445. That is 26.9% less code for the same 4,444 tasks.

It used 50,077 functions to do that, down from 65,513, a 23.6% reduction. Work that out per function and you get 9.6 lines against Astra's 10.0, so functions are marginally smaller too. This is less code at roughly the same granularity, not the same work packed into denser units.

Comment lines fell much further than the code did. Comment density is 1.3% of output, down from 4.2%, which in absolute terms is 6,219 comment lines against Astra's 28,824. Sol returns to the range the GPT-5.6 variants sat in, at 1.5% and 0.9%.

For anyone maintaining this code later, there is considerably less inline context than Astra provided.

Does GPT-6 Sol generate more complex code than Astra?

Both complexity measures moved up slightly, and by almost the same amount.

Cyclomatic complexity is 225.56 per kLOC, up from 220.93, a 2.1% increase. Cognitive complexity is 167.38 per kLOC, up from 163.38, a 2.4% increase.

At that size neither is a meaningful change on its own. What matters is the direction combined with the volume. Sol spreads slightly denser logic across 27% fewer lines, so the total amount of branching and nesting in the output came down substantially even though the per-line figures edged up.

What is GPT-6 Sol's bug density compared to GPT-6 Astra?

Bug density is 629 per mLOC, up from 558. A 12.8% increase.

The two severe tiers swapped places. Blocker bugs rose 51%, from 37 to 56 per mLOC. Critical bugs fell 37%, from 70 to 44. Major rose 31% and minor is essentially flat.

Astra concentrated its severe findings in the critical tier. Sol moves more of them up into blocker, which is the tier most likely to reach production. That is worth knowing even though the two roughly cancel in total.

In absolute terms the picture is friendlier, because there is 27% less code. Sol produced 302 bugs across the benchmark against Astra's 366, so 17.5% fewer bugs in total despite the higher rate per line.

Concurrency and threading is still the largest bug category in both variants, and it is the one that improved. Sol brings it down 12% by rate, from 276 to 242 per mLOC, and by 36% in absolute terms, from 181 findings to 116.

Exception handling improved too, down 51% by rate and from 44 findings to 16.

Most other categories rose by rate. Null and data value more than doubled, control flow more than doubled, type safety rose 79%, resource leaks 71%. Read the finding counts alongside them though. Resource leaks went from 25 findings to 31 and type safety from 19 to 25, so these are modest absolute movements on small bases amplified by a smaller denominator.

Concurrency findings stay the expensive kind regardless of the improvement. They are hard to reproduce, they depend on the environment they run in, and they surface as intermittent failures rather than clean ones. The categories that flag them are narrow and specific: double-checked locking, a lock not released on every path out of a method, synchronizing on a field that gets reassigned, a counter incremented on a volatile field, and calls to wait or notify made without holding the lock. Those turn up in singleton and cache setup, connection and worker pools, shared counters, and producer and consumer queues.

What security vulnerabilities does GPT-6 Sol generate in code?

This is the one area where writing less code did not mean finding less.

Vulnerability density is 285 per mLOC, up from 178. A 60% increase. And unlike bugs and code smells, the absolute count rose too, from 117 findings to 137, which is 17.1% more across 27% less code.

Blocker vulnerability density is 2 per mLOC, down from 3, and that is the lowest figure we have recorded from this family. Read those rates with care, because the underlying numbers are tiny: one blocker vulnerability in Sol's output and two in Astra's. Both variants are very clean at the top of the severity scale, and at that scale a single finding moves the rate by a whole point.

Everything below the blocker rose. Critical went up 39% to 198 per mLOC, major more than doubled, and minor rose 167%. Critical remains the dominant tier at 69% of security density, against 80% for Astra.

Cryptography misconfiguration is the largest category in both variants, and its rate rose 39% while the count stayed roughly constant. Insecure system resource handling did the same thing, 39% by rate and a similar number of findings. Those two categories held steady in absolute terms and only look like increases because the model improved in other areas.

The genuine increases are injection attack, from 10 findings to 18, and inadequate error handling, from 3 to 8. Both are small categories.

Cryptography misconfiguration and insecure system resource handling together account for 191 of the 284 per mLOC, so two thirds of the security surface sits in two categories. That has been true of every model in this family, and it is the answer to where a pipeline should point first. Crypto misconfiguration covers weak algorithms, insecure key sizes, and random number generators used in ways they should not be.

How maintainable is GPT-6 Sol generated code?

Code smell density is 19.62 per kLOC, up from 18.47. A 6.2% increase.

In absolute terms smells came down substantially, from 12,123 findings to 9,420, which is 22.3% fewer.

Blocker maintainability findings came down 12%. Minor came down 14%. Major rose 20% and carries most of the density increase.

This is the table where the two columns tell the clearest story. Every single category came down in finding count except deprecated API usage, which went from 112 to 142. Collection and generics fell from 5,243 findings to 4,138. Design and framework fell from 1,717 to 1,242. Regex and string formatting fell from 1,376 to 932.

By rate, most of those same categories rose, because the denominator shrank faster than the findings did. That is the honest reading of Sol's maintainability profile: less of everything, spread across less code, so the per-line figure edges up while the pile of work gets smaller.

Collection and generics is still the largest smell category by a wide margin in both variants. These are raw types where parameterized generics belong, and collection handling that sidesteps type safety. In Java they carry a real cost. Raw types bypass generic type checks and produce unchecked warnings, they make refactoring harder, and they defer the catch of unsafe code to runtime, where a properly typed implementation would have caught it at compile time. 

Deprecated API usage is the one to watch, up 73% by rate and up in count as well.

How does GPT-6 Sol code volume affect total findings?

Total findings fell 21.8%. Bugs fell 17.5% and code smells fell 22.3%. Vulnerabilities rose 17.1%.

Every density measure went the other way. Bug density up 12.8%, vulnerability density up 60%, code smell density up 6.2%, overall issue density up 6.9%.

So the two readings point in opposite directions on three of the four, and the gap between them is the 27% volume reduction. Density asks how clean the code is per line written. Absolute counts ask how much work exists. For a team sizing review effort, Sol produces less work: 480,000 lines instead of 656,000, and 9,859 findings instead of 12,606. For a team comparing how carefully each line needs reading, Astra is the cleaner output.

Vulnerabilities are the exception worth pausing on, because they are the one category where both readings agree. Sol produced more of them in absolute terms across less code. That is not a denominator effect and it is the single clearest argument for Astra between the two variants.

How do GPT-6 Sol and GPT-6 Astra compare on token usage?

Input tokens were identical across both runs at 1,322,069, which is what you would expect from models using the same tokenizer.

Output tokens came down 5.1%, from 6.65 million for Astra to 6.30 million for Sol. Set against 26.9% less code, that means Sol spent more tokens per line of output than Astra did.

Reasoning tokens are where the two diverge. Sol used 2.64 million against Astra's 1.36 million, so it nearly doubled its reasoning budget while producing slightly less output. Total tokens still came out 4.3% lower for Sol, at 7.63 million against 7.97 million, because the reasoning increase is smaller in absolute terms than the code reduction.

More thinking, less writing. That is a reasonable description of what separates these two variants.

What are the biggest risks of using GPT-6 Sol generated code in production?

Sol is the more concise of the two GPT-6 variants and it produces less work overall. It also gives back some correctness and some per-line cleanliness to get there. Three places to point verification effort.

Security is first, and it is the one place where the two readings agree. Vulnerability density rose 60% and the absolute count rose 17% across 27% less code. Cryptography misconfiguration and insecure system resource handling account for two thirds of that surface, and both are well covered by automated analysis. Blocker findings are very low in both variants, at 2 per mLOC for Sol, so this is about the critical tier rather than the most severe one.

The two roughly cancel in total, but the mix moves toward the highest impact tier. If your quality gate weights blocker findings more heavily than critical, that shift will show up.

The correctness gap is third. 83.09% against 85.85% means roughly one in six solutions fails its tests rather than one in seven. On 544 test-backed tasks that is a difference of about 15 solutions. Whether that matters depends on how much you lean on the pass rate versus how much you value the smaller review surface.

Concurrency is the good news. It remains the largest bug category in both variants, and Sol improves it on both measures, down 12% by rate and down 36% in finding count.

Three takeaways:

• Sol is the concise variant. 26.9% less code than Astra for the same tasks, 23.6% fewer functions, and 21.8% fewer findings in total. Bugs and code smells both came down in absolute terms.
• Security is the exception. Vulnerabilities rose in both density and count, which makes them the one finding type where less code did not mean less to fix. Blocker findings stay very low at 2 per mLOC.
• The variants are a choice, not a ranking. Astra for the higher pass rate and the cleaner per-line profile. Sol for a smaller review surface and fewer findings to work through. Pick by whether your bottleneck is review depth or review volume.

Full evaluation results for GPT-6 Sol, along with every other model we have measured, are on the Sonar LLM Leaderboard.