Resolution: Won't Do
RSPEC has a field named "Analysis Scope" that specifies if a rule applies to main sources, test sources or both. This field is visible on SonarQube and on rules.sonarsource.com. However, this does not reflect exactly what is happening during scan, as the scope "ALL" is ignored by analyzers. Moreover in SonarC# and SonarJava the value of this RSPEC field is not used and the scope of each rule is currently hardcoded. So the RSPEC and the rule implementation in SonarC# and SonarJava can be out of sync. For all the other code analyzers, no rules are applied to test source files, which are often not even analyzed (no highlighting, no symbol table, no issues).
We do expect to get access to this "Analysis Scope" value in the description of a SQ rule and in rules.sonarsource.com. Moreover any change to this RSPEC field should be automatically propagated in the rule implementation, in the SQ rule description and in rules.sonarsource.com. Whereas currently only SonarJava and SonarC# are able to apply some rules to the test source files, we do expect all the code analyzers to support the analysis of test source files (See Open questions).
Regarding UI of SQ, we do expect to have a dedicated facets relying/benefiting on this scope:
- new facet for scope on rule page
- maybe new facet for file type on issues page
- it should be more obvious from UI point of view if file is MAIN or TEST (currently there is only small icon without clear semantic)
This scope field has been added to rule-api (in the rule-metadata-schema.json). Its name is "scope" and its value can be a string with 3 possible values "Main", "Tests" and "All".For backward compatibility if the field is absent in the RSPEC, the default value should be "Main".
This field should now be used by:
- Analyzers to automatically run the rules on the appropriate scope: SonarJava and SonarC# must be updated accordingly.
The scope has been added to the rule metadata schema. The schema for the field is:
Currently the SonarJava analyzer does not enforce that field.
- It uses a "tests" label for rules that will be run only on test code.
- Rules that don't have that label are never run on test code (but adding the label to a rule not having it has no effect) and rules which are targeting "both" are consequently only played against main code.
- Which rules should have the All scope?
- one way to go is to define this based on Sonar Way (only a subset of rules from SonarWay should have the both), but what about the other rules?
- Some analyzers are proposing to register custom rules (Java, C#, php, JS, etc...). On SonarJava for instance, custom plugins are expected to give list of rules targeting main sources, and another list of rules targeting tests.
- What should we do regarding the scope of these rules?
- Should we allow to use All ?
- The computation of debt is the ratio between number of Issue and number of LOC, but tests files are not included in computation of LOC.
- How to take (or not to take) into account these new issue in the metrics?
- What about computation of technical debt?
- What about the impact of the new feature on Quality Gates?
- Once the feature would have been enabled, it means that issues on test sources are most probably going to occur more often. Possible consequences are quality gate failing, and significant increase of debt. Should we provide a way to ignore issues on tests on computation of debt/quality gate?
- We should think about SQ compatibility. One way to go, if the is no scope in SQ (<7.1), then rules targeting both scopes, should create issues only for main.
Currently we have a set of metrics depending on issues (debt or number):
- Reliability (bugs number)
- Maintainability (code smells number, debt, debt ratio)
Bugs number and maintainability debt ratio are used by default for a quality gate, thus the way we count them might greatly influence the project quality view. Currently it's already possible to have issues on test files, and we can already observe the inconsistency of metrics:
- Bugs number is an aggregation of bugs on Main and Tests files. (Bugs = Bugs Main + Bugs Tests)
- Debt ratio is a ratio of debt coming from both Main and Tests files to lines of code of Main files only (DebtRatio = (CS Main + CS Tests)/LOC Main*k )
So we have 2 problems
- Metrics are inconsistent (for DebtRatio): issues coming from both scopes while lines of code are only for Main.
- We can easily imagine both someone willing to assert the quality of Tests and someone not willing to do that.
We should have separate sets of metrics for Main and Tests files, all of which could be used as a quality gate condition.
- Bugs (for Main)
- Bugs on Tests
- Code Smells (CS), dept, dept ratio (DebtRatio = CS Main/LOC Main*k) (for Main)
- Code Smells on Tests, dept on Tests, dept ratio on Tests (DebtRatio = CS Tests/LOC Tests*k')
We might extend Project Page to add new block showing the status of test files quality (see picture attached).