Uploaded image for project: 'Product Roadmaps'
  1. Product Roadmaps
  2. MMF-1369

Real Quality Gates for PRs and short-lived branches

    XMLWordPrintable

    Details

      Description

      WHY

      We currently impose a very simple, hard-coded "Quality Gate" on PRs and short-lived branches: No open issues. This is overly simplistic and not in keeping with how Quality Gates work.

      With the addition of Coverage (MMF-1118) and Duplication (MMF-1396) metrics, it becomes possible to have a more sophisticated status which would really show if the quality of the code is good enough for a PR/short-lived branch to be merged.

      WHAT

      Status
      • We want the status on PRs and short-lived branches to correspond to the quality of the code that is added/updated.
        We want developers to
        • focus on the code they are writing
        • make sure of the quality of the code they are pushing
      • As much as possible, we expect the status to indicate if merging the PR/short-lived branch will fail the Quality Gate of the long living branch.
      • But we don't try to see what will be the Quality Gate status of the target branch once the code is merged.
      Conditions

      As a consequence, we want conditions on PRs and short-lived branches to be similar to the Quality Gate of the main branch, but only "on new code" conditions should apply:

      • "new code" measures already computed and displayed on PRs/short-lived branches:
        • new open (not confirmed) bugs, vulnerabilities and code smells
        • coverage ratio on new code, new lines to cover
        • duplication ratio on new code, new duplicated lines, etc
      • but also any other "new code" measures that can be used as conditions in the Quality Gate definition:
        • reliability, security and maintainability ratings on new code
        • technical debt ratio on new code, reliability remediation effort on new code, etc

      Even if the number of changed lines is small, we should in general be able to apply those conditions.
      But, as for long-lived branches, we will apply a threshold: conditions on coverage and duplication should be ignored if less than 20 lines were updated. When it's the case, an explanation message should be displayed close to the status (see SONAR-9352 for the details):

      Some Quality Gate conditions on New Code were ignored because of the small number of New Lines

      Note:
      The Sonar way Quality Gate definition doesn't exclude confirmed issues from the issues taken into account to compute the the Quality Gate status. Then, by default, if some issues are detected in a PR, the only way to by-pass the red status is to mark these issues as won't fix/false positive.
      Also, there will be no inconsistency anymore between the issues counts displayed in SonarQube/SonarCloud and the PR status.

      Measures

      As with the long living branches, the Measures page on PRs and short-lived branches should display the main measures on new code:

      • Reliability: bugs, rating, remediation effort
      • Security: vulnerabilities, rating, remediation effort
      • Maintainability: code smells, debt, debt ratio, rating
      • Size: new lines
      • Issues: new issues
      If no SCM data

      PR concept refers to Git, and in this case we have SCM data we can rely on.
      But short lived-branches can be created whereas SCM data is unavailable (SCM is not supported). In this case, to identify which piece of code is new, we compare the code with the long-living branch and apply a differential algorithm to track changed files (MMF-808). As a consequence, as for the master, we'll should still be able to apply "new code" criteria to compute short-lived branch status.

      HOW

      List of PRs
      • The status for PRs/short-lived branches will now correspond to the Quality Gate status and will be displayed with the same visual indicator than for all the branches. The current indicator can be replaced by the Quality Gate badge.
      • Issues counts won't be displayed anymore to not let think that this is the only condition that affects the status,.
      Branches & Pull Requests management
      • Here again, the Quality Gate badge will replace the old status and the number of issues.
      PRs/short-lived branches pages

      The different conditions that contribute to the Quality Gate status won't be displayed in the header.

      A new Overview tab will appear for PRs/short-lived branches, with:

      • The Quality Gate status (the information could possibly be repeated in all pages)
      • Conditions in failure
      • Main metrics on new code to help to understand the status of the PR:
        • Reliability, security and maintainability ratings
        • Number of bugs, vulnerabilities and code smells
        • Coverage %
        • Duplications %
        • New lines to cover
        • New lines of code
      • Started at
      • A link to the PR on the ALM (See the PR)
      • Explanations about the Quality Gate that is used, and about the fact that only conditions on new code are evaluated
      • Coverage and duplications estimated after merge, which will move from Measures page to the Overview

      The 'new code' yellow background won't be used in this overview. It will also disappear from Measures and Code pages for PRs/short-lived branches since almost all figures in those pages is about new code. Still, it can be useful to explain that most metrics are on new/updated code.

      ALM integration
      • AzureDevops (SonarCloud) / TFS (SonarQube)
        The Code quality check sum up, in the Overview of the PR, will stop showing the number of issues, coverage and duplication.
        Only the status will be displayed, with a link to SonarQube/SonarCloud to provide more details. The same applies to the Report build status step of the build pipeline.
      • BitBucket Cloud (SonarCloud only)
        • Builds will stop displaying the number of issues, coverage and duplication
        • Code Quality in Overview will show
          • status success/failed
          • number of bugs, vulnerabilities, code smells, coverage, duplications
          • a link to see more details on SonarQube/SonarCloud
      • GitHub (SonarCloud) / GitHub Enterprise (SonarQube)
        • Code Analysis sum up in PR Conversation will stop displaying the number of issues, coverage and duplication
        • Comments (SonarQube only) in PR Conversation will continue displaying issues only.
        • Checks (SonarCloud only) will show:
          • status success/failed
          • number of bugs, vulnerabilities, code smells, coverage, duplications
          • a link to see more details on SonarQube/SonarCloud

      Current design solution

      Overview dashboard

      In Failed state:

      Behavior:

      • The first 5 Failed conditions boxes will appear by default. If there are more, they'll be hidden under a toggle link
      • Failed conditions corresponding to our Built-in QP should appear first
      • They should be ordered in the same way than on the Metrics block on the right:
        • Reliability
        • Security
        • Maintainability
        • Coverage
        • Duplications

      In Passed state:

      GitHub Checks

      On the PR:

      The check will simply display "Quality Gate: Failed" or "Quality Gate: Passed". For more info, the user should click on "Details"

      On a failed check:

      • The list of failed conditions appears under the Quality Gate badge. Even if those conditions do not concern issues, coverage or duplications (example here with Cognitive Complexity condition)
      • Issues should be displayed with their rating badge
      • Coverage and Duplications should also display the estimates

      On a failed check with no coverage information:

      On a failed check with no issues:

      On a failed analysis:

      BitBucket PR Decoration

      When the Quality Gate is Passed

      When the Quality Gate is Failed

      Failed conditions appear first, and the rest is toggled to keep the user's attention on what's failed.

      Once toggled, it looks like this:

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              christophe.levis Christophe Levis
              Reporter:
              ann.campbell.2 Ann Campbell
              Votes:
              30 Vote for this issue
              Watchers:
              39 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: