TL;DR overview
- This update to Sonar's Coding Personalities of Leading LLMs report adds GPT-5 analysis data, evaluating the model's code quality output using SonarQube Enterprise static analysis on Java programming tasks.
- GPT-5 demonstrates improved benchmark pass rates compared to earlier GPT versions, but like other models in the study, produces code where code smells constitute over 90% of all detected issues, reflecting a shared tendency toward technically functional but structurally poor code.
- The update reinforces the core finding across all evaluated models: LLM upgrades do not consistently translate to better production-ready code quality—teams must apply automated static analysis to LLM output regardless of model generation.
- SonarQube users can rely on consistent, model-agnostic quality and security verification across all LLMs their teams use, ensuring AI-generated code meets the same standards as human-written code.
In our previous report, “The Coding Personalities of Leading LLMs,” we revealed the shared strengths and flaws of some of the most popular LLMs, while also uncovering distinct coding “personalities” for each model.
GPT-5’s arrival on the scene adds an important new dimension to the landscape, so we have updated our analysis to include it. To do an apples-to-apples comparison, we evaluated GPT-5 with minimal reasoning against Anthropic's Claude Sonnet 4 and 3.7, OpenAI's GPT-4o, Meta's Llama 3.2 90B, and the open source OpenCoder-8B.
Bottom line: GPT-5-minimal reasoning does not unseat Claude Sonnet 4 as the performance leader. It performs better than every other model we tested, but had lower functional performance than Claude Sonnet 4, while generating more verbose, complex, and issue-prone code. Claude Sonnet 4 remains the leader of the non-reasoning models, both in terms of functional performance and code quality.
A note on methodology
Using the SonarQube Enterprise static analysis engine, Sonar has now evaluated code generated from six leading LLMs, including the latest GPT-5 model from OpenAI. Each model was tested against over 4,400 unique Java assignments from recognized benchmarks like MultiPL-E and ComplexCodeEval.
For this evaluation, we analyzed “GPT-5-minimal,” which operates at the model’s lowest reasoning level, to have a fair comparison with other models like Claude Sonnet 4 that have reasoning disabled by default. Reasoning adds a number of dimensions to this analysis, which we will explore in future work.
Functional performance
The first dimension of any model’s personality is its raw functional skill. On this front, GPT-5-minimal establishes itself as a highly-competitive model with a weighted pass average of ~75%. GPT-5-minimal is a top-tier performer, second only to Claude Sonnet 4.
Table 1: Functional performance on MultiPL-E Java benchmarks
The cost of performance: Extreme verbosity and complexity
We have previously seen that models that do well functionally tend to generate more lines of code per completed task. GPT-5 breaks this trend; despite not being the top performer, GPT-5-minimal generates a substantially larger and more complex volume of code than any other model, including Claude Sonnet 4.
Table 2: Code volume and complexity metrics
GPT-5-minimal produced 490,010 lines of code, over 30% more than the top-performing Claude Sonnet 4. The code it generates also has a dramatic increase in cyclomatic complexity and cognitive complexity. Developers who need to review code generated by GPT-5-minimal will be faced with a tough challenge.
Deep dive into code quality
Compounding these challenges, we find that the code from GPT-5-minimal has a much higher density of issues relative to the tasks it solves.
Table 3 highlights the “Issues per passing task” from each model. GPT-5-minimal introduces 3.90 issues for every correct solution—nearly double the rate of the more concise and higher performing Claude Sonnet 4. This means for every task it completes successfully, it introduces significantly more potential defects than its competitors, resulting in a large downstream technical debt, quality, security, and verification burden.
Table 3: Overall code quality and issue rates
GPT-5-minimal produces the lowest density of vulnerabilities by a wide margin (0.12 per KLOC) and a relatively low bug density. However, this is balanced by a much higher density of code smells (25.28 per KLOC), indicating a primary weakness in code quality and maintainability.
Table 4: Issue density by type (per KLOC)
While a low density can sometimes be misleading if a model is simply more verbose, the data on absolute vulnerability counts confirms this is not the case. With only 60 total vulnerabilities generated, GPT-5-minimal is proving its security focus is strong on both a relative and absolute basis.
Table 5: Absolute vulnerability counts
Here is some more detail regarding the coding personality of GPT-5-minimal.
Security
GPT-5-minimal’s strongest trait is its focus on security, which is evident across multiple metrics. Its issues are far less likely to be security-related; only 0.46% of its total discovered issues are vulnerabilities, a fraction of the rate for other models. Furthermore, it produces the lowest density of vulnerabilities of any model tested—just 0.12 per 1,000 lines of code (KLOC).
However, there are caveats too. The model shows a tendency to reintroduce classic security flaws that are less common in other models. Key issues include Path-traversal & Injection flaws that make up 20% of its total security vulnerabilities, indicating a different and more fundamental risk profile.
Maintainability
This model’s strong security performance is balanced by its weaker performance on code quality and maintainability.
- Code smell density: Its code is inherently less maintainable, with a high density of ~25 code smells per 1,000 lines of code.
- Complex issues: The core issue is the code’s intricacy. The solutions generated by GPT-5-minimal result in the highest percentage of code smells related to “Cognitive/computational complexity” (~12%) among all evaluated models. This tendency to produce overly complex code directly creates long-term technical debt, making it difficult to understand and maintain in the future.
Reliability
This model demonstrates a higher rate of foundational logical errors compared to its peers. “Control-flow mistake” bugs are dominant, accounting for roughly 24% of its total functional bugs. This shows that while the model can produce functionally-correct code, it is also prone to making basic logical errors.
Conclusion: Trust and verify
GPT-5 is undeniably a powerful new force in AI code generation. However, this analysis of its minimal reasoning mode shows that progress is not linear. It reveals a model that, while functionally proficient, carries a significant quality cost and presents a different profile of security and reliability considerations.
This makes the “trust and verify” mandate more critical than ever. To leverage this model's power, organizations must evolve their governance strategies:
- Manage the complexity: Its code is a prime candidate for refactoring. Static analysis is essential to identify the critical code smells and high-complexity methods that will quickly become unmaintainable.
- Scrutinize for advanced flaws: Code reviewers must be vigilant for this model's specific tendencies, including the re-emergence of classic vulnerabilities like path-traversal and a higher rate of fundamental logic errors.
This analysis shows that as AI models evolve, their flaw profiles become more nuanced. Harnessing their potential requires an equally sophisticated and adaptable approach to governance.