The current management of coverage in SQ has a number of problems:
- We're dealing with multiple coverage tools that provide non-homogeneous data. Some provide only aggregate, project-level coverage metrics; some provide file-level coverage data; and others provide line-by-line coverage data. Calculating project coverage across multiple languages that use tools that provide disparate data sets is a nightmare.
- How do you compute Overall Coverage for conditions? If the Unit Test data says 2 of 4 conditions on a line are covered, and the Integration Test data says 2 of 4 are covered... is that a total coverage of 2, 3, or 4 of 4?
- How do you compute coverage for a project when the report(s) only contain data on a subset of files? Are they missing from the data because they're not executable or because they're not covered at all and were 'missed' by the coverage engine?
- How do you recognize which files are "testable" in order to force overage to 0 on them? E.G. not all .java files with nLoC are testable (interfaces)
- Some projects only have integration tests; it is their primary means of testing. But Integration Tests are treated differently in terms of metric calculation than Unit tests. How to have a single QG? Solution would be Overall coverage but we are back to aggregation issue. BTW overall coverage is no more displayed in UI.
Drop concepts of "Unit coverage", "Integration coverage", "Overall coverage". Now there is just "coverage".
Coverage is additive. If tool A reports coverage on lines 4, and 6, and tool B reports coverage on lines 3, 7, 9, then the file has 5 covered lines. We don't care how they're covered (unit, integration, smoke, etc)
When multiple tools report condition coverage on a line, the handling of that corner case may result in a pessimistic view of the coverage, but is guaranteed not to result in an overly-optimistic view:
- different numbers of hits: 2 of 4, 1 of 4, 3 of 4, we take the max => 3 of 4
it may be that 4 of 4 are covered, but all we know for sure is that 3 of 4 are covered
- different denominators (extremely pointy corner case): 1 of 3, 1 of 4, 2 of 4, we take the larger denominator => 2 of 4
When multiple tools are used that report disparate data types (corner case), e.g. file-global coverage numbers + line-by-line hits, the platform will use line-by-line hits and ignore the file-global data. File-global data will be used if that's all there is. If multiple types of file-global data are submitted, we'll take the biggest number.
Fall back on the executable lines reported by language plugin IFF no tools reported coverage for the file. (This yields "force to 0" behavior).
We no longer accept project-global coverage data. Only at the line or file level will data be accepted. If it still exists, API support for submitting project-level coverage data will go away
- deprecate coverage type parameter in API. Internally it will be ignored
- accept the same API being called multiple times/file
- rename "Unit test*" metrics to "Coverage*". This will maintain a good history for most people. Those with Unit + IT will loose some data
- "Integration test*", "Overall Coverage*" metrics will be dropped.
- UI to be updated accordingly. This can happen asynchronously
- Update quality gate conditions on Overall Coverage metrics to be against the relevant Coverage metric
- What if user has conditions on both Overall and Unit Test coverage?
Plugins will need to offer new properties to fully support this. For instance the Java analyzer should replace
Thanks to the API that allows plugins to know in which SQ version they run, they will be able to correctly handle the different cases. For instance:
- throwing an error message if one is using the new property while running a 5.6 LTS instance
- if one is still using the old properties, doing the right API calls depending on the version of SQ
Plugins should also begin reporting executable_lines in a file, to indicate which lines should be tested. This data will be used to force coverage to zero when no tests are reported for a file. executable_lines is required, as opposed to using ncloc because not every statement is executable. For example, we don't want to show 0 coverage on a Java interface.
Beyond that, the Generic Coverage plugin should be updated as well to handle multiple coverage reports.