TL;DR overview
- Code coverage is a software testing metric that measures the percentage of source code executed during testing, revealing untested sections.
- Common coverage types include line, branch, function, and statement coverage, each measuring a different dimension of test thoroughness.
- Code coverage testing involves identifying code to test, preparing a test suite, running it, and analyzing the resulting data.
- SonarQube integrates coverage reports to visualize gaps and enforce coverage thresholds through quality gates.
What is code coverage?
Code coverage is a metric used in software testing to measure the degree to which the source code of a program is executed during testing. It aids in identifying the extent to which the source code is being exercised, allowing you to acquire a better awareness of your testing efforts and where extra testing may be required.
The goal of code coverage is to discover sections of a program that are not being exercised by a set of test cases; therefore guiding the development of new tests to obtain better coverage.
The term "code coverage" derives from the field of software testing, where it is used to determine which sections of the code are covered (tested) and which are not. This data is quite helpful in understanding the quality of the test cases and the comprehensiveness of the testing.
What is code coverage testing?
Code coverage testing is a type of software testing that determines the extent to which your test executes your source code. The major purpose of code coverage testing is to identify sections that have not been tested and to guarantee that the application's essential paths have been tested.
Identify the code that needs to be tested
First, you need to identify the source code that must be tested which could be the whole application or just a particular module.
Prepare the test suite
Then, the test suite is prepared that includes all of the test cases that will be used to test the identified code. Unit tests, integration tests, and system tests may all be included in this step.
Run the test
The test suite is executed against the identified code. As the test suite runs, various components of the code run.
Analyze the data coverage
After the test suite has been completed, the coverage data is examined. This could include producing a coverage report that reveals which sections of the code were it was and was not executed.
Improve the test suite
If the coverage data shows that some sections of the code weren't carried out by the test suite, more tests can be developed to cover those sections. The goal is to gradually improve the test suite in order to increase code coverage.
What is a good code coverage percentage?
While there is no consensus on what is an acceptable code coverage percentage, the typically accepted goal for code coverage is around 80% to target.
There are several reasons why 80% coverage is regarded as a reasonable goal. For starters, it strikes a fair balance between coverage, benefits and price. Attempting to obtain 100% coverage can be costly and time-consuming, and it may not be essential to achieve that degree of coverage to detect the majority of issues in your code. Second, 80% coverage should provide you with a high level of confidence that your code is working properly.
For some projects, 80% coverage may not be sufficient. For example, if your project is vital for your company's success or contains a lot of sophisticated logic, you should shoot for a higher level of coverage.
A code coverage percentage is a measurement that represents the percentage of code that has been run during testing. It reflects how thoroughly the test suite has run through the source code. A higher percentage of code coverage generally indicates greater testing thoroughness, but it does not guarantee the absence of errors or complete test coverage.
A code coverage percentage measures the extent to which the source code was executed during testing. While it is a useful measure of test coverage, it should be used in conjunction with other testing procedures to achieve thorough testing and excellent software quality.
Additional variables to consider when deciding on your code coverage target:
- Your code’s complexity.
- The significance of your project.
- Your financial circumstances and available resources.
- The testing experience of your team.
Why is code coverage important?
Good coverage offers developers reassurance that their entire project is properly built and maintained, making it easy to assess whether or not a recent modification or addition to the source code will fail. Code coverage is important since it can help identify portions of your code that are not being tested. This can help detect flaws and ensure the proper operation of your code.
Code coverage identifies sections of code that were not performed during testing such as; highlighting potential gaps in test coverage. Performing coverage tests allows for improvements to software quality by providing a quantitative assessment of how much the codebase was explored.
How does code coverage work?
Code coverage works by measuring the amount to which a program’s source code is executed during testing. This includes tracking which parts of the code are run and which are not. Code coverage is typically measured using a code coverage tool.
The steps can be divided into the following steps:
Instrumentation
Your code has instrumentation inserted into it by a code coverage tool, which is small enough not to affect code performance. During instrumentation, the code is updated or enhanced to incorporate additional instructions that record information about its execution. These specific code blocks, statements, branches, or functions are often logged.
Test Execution
As the test is executed, the code coverage tool gathers the data about the executed code. As the test runs, the code inserted during instrumentation verifies the functionality of the software application.
Data Collection
During this step, the data is collected about which parts of your code have been executed by your tests. Data may include information such as the number of times a line of code is run, if a branch was taken or not, or whether a function was called.
Coverage Metrics
The data gathered is then used to determine various coverage metrics to check how much of your source code was executed during testing.
Reporting
Reporting is the process of generating compilations of data that show which parts of your code have been executed by your tests. These compilations normally provide a code summary, mostly shown as a percentage, along with the details of the particular areas of code that have been covered.
Analysis
The code coverage report is analyzed to discover portions of the code that have inadequate or no overage. This analysis assists developers in increasing test coverage by identifying potential gaps.
Action
After the analysis is completed, the developer can take action by adding new tests, modifying existing tests, or changing the testing method to improve code coverage.
What are code coverage types?
Code coverage types are different dimensions or features of the code that can be assessed and reviewed during the code coverage analysis. These types assist in providing a thorough knowledge of the coverage attained during testing.
Below are the most common code coverage types:
Line coverage
Line coverage is the percentage of lines of code that have been run during testing. It checks to see if each line of code has been executed at least once. This is one of the most fundamental types of coverage.
Statement Coverage
Statement coverage is closely related to line coverage, however, it focuses on individual statements instead of lines. It assesses code coverage at a finer granularity than line coverage.
Branch Coverage
Branch coverage determines whether or not every conceivable branch or decision point in the code has been performed. It determines whether or not all conceivable outcomes of conditional statements, loops, and switch cases have been tried.
Function/Method Coverage
Function (or method) coverage emphasizes that each function or method in the code has been used at least once during testing.
The reason is that there may be multiple functions depending on the input and the functions may or may not be called. This goal is to ensure there is a function that is able to be called.
Condition Coverage
Condition coverage measures the percentage of conditions in your code that are executed by tests. Condition coverage is much more granular than branch coverage as it prioritizes specific conditions within branches.
Path Coverage
Path coverage is the most exhaustive, but also the most difficult type of coverage to obtain. Throughout testing, it determines whether each conceivable execution path through the code has been visited.
These code coverage types give varying levels of detail and concentrate on distinct elements of code execution. They aid in the identification of untested areas and the enhancement of test suites. Developers can acquire a more comprehensive view of the coverage achieved and discover areas that require extra testing by combining various coverage types.
Code coverage tools
Code coverage tools are software tools or frameworks that help in measuring and analyzing code coverage during software testing. These tools aid in the automation of the process of gathering coverage data, generating coverage reports, and assessing the test suite's coverage. They give developers insight into which parts of the code have been tested and which have yet to be tested.
Some popular code coverage tools include:
- JaCoCo (Java)
- Cobertura (Java)
- Istanbul (JavaScript/TypeScript)
- Coverage.py (Python)
- gcov (C/C++)
Overall, code coverage tools can be a valuable tool for improving the quality of your software.
Sonar and code coverage
Sonar’s SonarQube Server and SonarQube Cloud solutions act as a central hub that integrates with popular code coverage tools. It consolidates coverage reports with static code analysis results, giving you clear pass/fail metrics to ensure your code meets quality standards efficiently.
Sonar offers a variety of code coverage metrics enabling organizations to gain intricate insight into the extent of test coverage on their code.
Sonar easily integrates into existing development workflows, facilitating organizations in measuring code coverage without derailing their established processes.
Sonar provides solutions that can be used by developers, QA engineers, and other stakeholders.
Sonar creates powerful, insightful reports and dashboards, facilitating organizations' communication of the status of code coverage to stakeholders.