Sonar put Claude LLM models to the test and unveiled suprising insights. Explore the habits, blind spots, and arcetypes of Claude LLM models to uncover considerations for your codebase.
Our analysis shows that each Claude LLM model has a unique and measurable coding personality. Which one have you "hired" for your team?
Claude Sonnet 4
Senior architect
This LLM codes like a seasoned and ambitious architect tasked with building enterprise-grade systems. It exhibits the highest functional skill, successfully passing 77.04% of the benchmark tests. Its style is verbose and highly complex, as it consistently attempts to implement sophisticated safeguards, error handling, and advanced features, mirroring the behavior of a senior engineer.
This very sophistication creates a trap: teams may feel the code is safer because it looks advanced, while in reality it likely introduces more complex, high-severity bugs like resource leaks.
The very sophistication of the model creates a lot of opportunities for higher- risk bugs that plague complex, stateful systems. Its unique bug profile reveals a high propensity for difficult concurrency and threading bugs (9.81% of its total bugs) and a significant rate of resource management leaks (15.07% of its bugs). The model's strength—its focus on generating sophisticated code—is linked to its weakness.
Claude 3.7 Sonnet
Balanced predecessor
This model represents a capable and well-rounded developer from a prior generation, exhibiting strong functional skills with a 72.46% benchmark pass rate. Its most defining personality trait is its communication style—it is an exceptional documentarian, producing code with a remarkable 16.4% comment density—nearly three times higher than its successor and the highest of any model evaluated. This makes its code uniquely readable and easier for human developers to understand.
But here's the catch with the balanced predecessor: while it appears more stable and less reckless than its more ambitious successor, it is by no means a “safe” model. It still introduces a high proportion of ‘BLOCKER’ vulnerabilities (56.03%) and suffers from the same foundational flaws as the other models.
What our analysis uncovered
%
more likely for new Claude model to be of 'BLOCKER' severity than its predecessor.
%
of all issues found in LLM-generated code create long-term technical debt.
%
of the vulnerabilities for one LLM are of ‘BLOCKER’ severity.
%
of all bugs from one popular LLM are control-flow mistakes.
Claude's bias towards messy code
Perhaps the most fundamental shared flaw of LLMs analyzed are a deep, inherent tendency towards producing “messy” code. For all non-reasoning models, code smells are the vast majority of issues. The following table provides the most direct evidence of the maintainability trade-off.
Sonar classifies the issues found in every project or codebase across three deeply interconnected software qualities: reliability, security, and maintainability.
Reliability
Bugs that would affect the software's capability to maintain its level of performance under promised conditions, potentially compromising its reliability and operational effectiveness.
Vulnerabilities and security hotspots. Vulnerabilities are code weaknesses that could be exploited for attacks, while hotspots are security-sensitive code requiring manual review.
Code smells, which could indicate weaknesses in design that can increase technical debt, slow down development, or increase the risk of bugs or failures down the line.
