TL;DR overview
- Unit test execution data in SonarQube provides visibility into test coverage, test success rates, and test duration across the entire codebase.
- SonarQube imports test results from external testing frameworks and correlates them with code analysis to identify untested code paths and quality gate violations.
- Coverage metrics combined with quality gates enforce minimum test thresholds on new code, ensuring incremental improvement without requiring retroactive coverage of legacy code.
- Integrating test execution data into SonarQube analysis gives teams a unified view of code quality, security, and test health in a single dashboard.
Starting with Java Ecosystem version 2.2 (compatible with SonarQube Server version 4.2+), we no longer drive the execution of unit tests during Maven analysis. Dropping this feature seemed like such a natural step to us that we were a little surprised when people asked us why we'd taken it.
Contrary to popular belief we didn't drop test execution simply to mess with people. :-) Actually, we've been on this path for a while now. We had previously dropped test execution during PHP and .NET analyses, so this Java-only, Maven-only execution was the last holdout. But that's trivial as a reason. Actually, it's something we never should have done in the first place.
In the early days of SonarQube Server, there was a focus on Maven for analysis, and an attempt to add all the bells and whistles. From a functional point of view, the execution of tests is something that never belonged to the analysis step; we just did it because we could. But really, it's the development team's responsibility to provide test execution reports. Because of the potential for conflicts among testing tools, the dev team are the only ones who truly know how to correctly execute a project's test suite. And in the words of SonarSource co-founder and CEO, Olivier Gaudin, "it was pretentious of us to think that we'd be able to master this in all cases."
And master it, we did not. So there we were, left supporting a misguided, gratuitous feature that we weren't sure we had full test coverage on. There are so many different, complex surefire configuration cases to cover that we just couldn't be sure we'd implemented tests for all of them.
Plus, This automated test execution during Java/Maven analysis had an ugly technical underbelly. It was the last thing standing in the way of removing some crufty, thorn-in-the-side, old code that we really needed to get rid of in order to be able to move forward efficiently. It had to go.
We realize that switching from test execution during analysis to test execution before analysis is a change, but it shouldn't be an onerous one. You simply go from
mvn clean install
mvn sonar:sonar
to
mvn clean org.jacoco:jacoco-maven-plugin:prepare-agent install -Dmaven.test.failure.ignore=true
mvn sonar:sonar
Your analysis will show the same results as before, and we're left with a cleaner code base that's easier to evolve.