Uploaded image for project: 'Minimal Marketable Features'
  1. Minimal Marketable Features
  2. MMF-365

Drop modules in SonarQube and drill down in the directory structure

    Details

    • Type: MMF
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Labels:

      Description

      WHY

      SonarQube modules are directly inherited from the Maven modules at a time where only the Java programming language and the Maven build system were supported by SonarQube.

      This SonarQube module concept provides a logical representation of a technical project which can greatly differ from the physical representation in the filesystem (directory/file). However, this advanced capability to be in abstraction with the physical representation has some major downsides:

      • In any source code repository and source code management service (GitHub, GitLab, Bitbucket, ...) we're used to browse the physical representation and to switch from SQ/SC to GitHub, Subversion, ... can be disruptive.
      • This logical representation is also used in .Net by creating one SonarQube module for each Visual Studio project. But at the opposite and by default the Scanner CLI provides a physical representation of the source code. This leads to some kind of inconsistency in term of user experience when moving from one language to another.
      • When working with a logical representation you can end-up with the same physical source file duplicated in different logical modules. In such case, all the metrics and issues for that source file are also duplicated.
      • At the opposite, the logical representation might not "index" all the existing source files and in such case those source files are not analyzed by SQ/SC. This limitation is particularly annoying in C/C++ (see SUPPORT-6614).
      • This concept was not designed to support nested modules whose parent modules contain themselves some source files.
      • This complicates the user experience for the following reasons:
        • Some settings are available at module level but others are not. Trying to understand the reasoning is impossible for end users. For instance, there is no longer any concept of QG/QP at module level whereas you can still define some file exclusions. Why?
        • At module level or for each directory containing some source files you retrieve a SQ/SC dashboard similar to the project dashboard and this prevents us from aggressively update the project dashboard. Indeed the constraint on both sides are not anymore the same as there is no QG, QP, branches, PR, ... at module level.
      • To activate the SonarLint Connected mode, it is currently required to select the appropriate module instead of just selecting the overall project and asking SonarLint to automatically infer what is the mapping between the local source directory and the remote one stored in SQ/SC.

      So at the end, we came to the agreement than Less Is More and it's time in SQ/SC/SL:

      • To stop supporting the concept of module
      • To display a physical representation of the project files
      • To drop the dashboards at module and directory level

      WHAT

      The idea is to completely drop the concept of module in SQ/SC and to make all the scanners (CLI, Maven, Gradle and MSBuil) feed SQ/SC with the physical representation of a source directory starting from ONE root directory. As a side effect, the dashboard visualization at package level will also vanish.

      Once that will be done, we do think that something will be missing: the ability to progressively drill down in the directory structure. This feature is really important to quickly understand, for instance, the breakdown of the code coverage at each directory level.

      From there, we could also start thinking about a way to group physical directories into some architectural layers but that's out of the scope for this MMF.

      HOW

      User Interface

      Measures page

      • Breakdown of coverage per folder:
        By clicking on the coverage rate in the project dashboard, the user should to be able to explore the tree of folders of the project in the same way they can browse the project in GitHub or in the file system.
        By default, folders and files should be sorted by alphabetical order instead of being sorted by coverage value.
        And, for each folder, the user should see:
        • the coverage rate that corresponds to all the files and folders under it;
        • the number of uncovered lines and conditions in all the files and folders under it;
      • More generally, in order to easily understand where the code can be improved, the user should be able to explore any measure this way.
        Ex:
        • Maintainability rating
        • Density of duplication on new code
      • Since we do expect this new visualization to be much more user-friendly, it should become the default mode when taking a look at the measures.
      • When exploring the tree of folders, figures in the facets should refresh to correspond to the folder that is currently displayed.
      • When a folder doesn't contain any file but only a single sub-folder, the user should directly see the sub-folder and the corresponding metrics (it should be done recursively).
        Ex: src/main/java/org/sonar/application
      • Files and folders should be presented with their relative paths to the project base directory.
        Ex:
        common/
        config/
        src/main/java/org/sonarqube/gradle/
        pom.xml
      • The breadcrumb should allow to quickly navigate through the different folders of the hierarchy. It could be done by displaying at least the top parent as well as the 3 direct parents of the displayed folder/file.
        Ex: SonarQube/sonar-scanner-engine/.../sonar/ce/queue
        The full path can appear and be actionable when the user hover the mouse over the breadcrumb.
      • The treemap should bring similar user experience: a treemap should show the metric per folder (considering all the files under the folder) and should allow to drill down, step after step, into the tree of folders.
      • The overview bubble chart provides a view at files level and is still relevant as it stands
      • Once on a specific folder, the user should be able to see the different metrics they are interested in.
        Ex: After looking at the coverage on new code, a user may want to take a look at the global coverage or even at the global duplication rate.
        For that purpose, the user should see the same view (same folder, with the same visualization) when selecting another metric in the facets, and it should not be possible to click again on the active facet.

      Code page
      The user should benefit from a similar user experience when using the code page: the user should to be able to explore the tree of folders of the project. And the breadcrumb should also act consistently with the measures page.

      Modules/folders spaces

      • Pages specific to modules and folders will be removed as of now.
      • Both at module and directory level, with it's currently possible to display measure history (in the activity page) and overview bubble chart (in the measures page).
        Those features will disappear and measure history should be purged.
      • The possibility to mark a module and a folder as favorite will vanish.

      Administration
      The possibility to update keys for modules won't make sense and will be removed

      Out of scope

      • We could think about having a merged experience for issues, measures, code and activity.
      • Sticky folders: Switching from measures to code, from a branch to another could keep the view on the open folder / file.

      Project analysis

      Scanners backward compatibility
      Scanner will no longer send information related to the modules but will report data about files only. The report sent to SQ/SC will no longer contain the file path that is relative to modules but only the path relative to the project base directory.
      Scanners such as the scanner for MSBuild or Maven are used to work with a modules hierarchy. The frontend will still rely on modules. The new scanner engine will be in charge of aggregating data for the different modules and will generate a report with files only.
      We can expect old scanners to remain compatible with future SQ/SC versions.
      Out of scope: Scanner for MSBuild frontend refers to files with their absolute path but provides a json report for each module. The scanner could provide a single report that would be the a merge for all modules so that modules disappear from the info provided to the scanner engine.

      Module/folder issues
      Our analyzers don't store measures on folders or modules, and currently only a few report issues on folders / modules. In fact, only SonarJava reports issues on folders (missing package-info.java). SonarC# reports some issues at modules level (about assemblies). Other Roslyn analyzers can only report issues at file and project level.
      We'll stop supporting issues at folder level as well as issues at module level.
      Issues that keep being reported on folders or on modules will be reported at root level. The info about the module/folder will automatically be appended to the issue description.

      Issues page

      • Module facet: Nowadays, even when there's no module in the project, the Module facet can display the root folder as a module when it contains files with issues. With the drop of modules, it doesn't make sense anymore to display this facet.
      • File facet: Now that we know for each intermediary folder the number of issues among the sub-folders, the File facet may become useless as it is and may need to be adapted / removed

      Settings

      1. SonarQube will stop supporting configuration defined in the UI at module level.
        A warning in SQ/SC project page will inform users that those settings won't be used anymore.
        Possible solution :
        • keep the old modules settings in a new project setting that is not used by analyzers.
          Ex: sonar.subproject.settings.archived=
          # Settings from 'SonarQube :: Core'
          sonar.coverage.exclusions=**/src/main/js/**,**/src/main/ts/**
          
          # Settings from 'SonarQube :: Scanner Engine'
          sonar.jacoco.reportPaths=target/jacoco.exec
          

          and make it clear that's it's only to show the configuration that was set on sub-projects, that it's deprecated as of now.

        • and report a warning on SQ/SC (MMF-1244) unless this setting is empty
      2. Scanners keep supporting settings defined at module level (in sonar-project.properties, pom.xml, csproj...).
      3. Settings defined at project root level will now be evaluated as paths relative to the project folder.
        Scanners will keep evaluating those settings as paths relative to the modules. If a file is excluded only with this second step, scanners and SQ/SC project page will display a warning, explaining that those settings should now contain paths relative to the project root.

      Out of scope

      • The Scanner for MSBuild supports some specific properties at module level. Those properties are not even configurable nowadays in SonarQube.
        Scanner for MSBuild could convert modules properties into solution settings.
      • Analyzers also support properties at module level. Analyzers could use global sensors and support properties at project root level.

      Migration
      When a project that was containing a module is analyzed again by SQ/SC, the first analysis will trigger a migration to clean up remaining modules information.

      • The Compute Engine will have to convert the reference to files from a path relative to the module into a path relative to the project (for each module, we know its relative path to the project).
        • Example of things to take into account: Detection of file move, loading SCM data from DB, issue tracker, ... All data loaded from DB might still contain modules in component keys.
      • Modules data including settings will be dropped from SQ/SC.

      Out of scope
      In the .Net world, MSBuild projects may not be in the solution folder. The case can be similar for a few maven projects for which the modules can be outside of the project folder. We already have a mechanism in that case to identify a common ancestor. We may need to adjust this mechanism to support the case where a new module would be added but would be outside this "root" folder.

      Migration path

      • Projects which are not analyzed again can still refer to modules. Module dashboards will be dropped, but SonarQube should still support browsing modules and folders for these projects.
        We'll keep the current way of browsing the project measures/code. We should then make clear that the figures displayed in that case are computed with the old style (no aggregation of children) and that, with the next analysis, modules will disappear and figures will be updated accordingly.
      • One day in the future (SonarQube 9.x), a migration should clean up projects that still contain modules.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                christophe.levis Christophe Levis
                Reporter:
                stas.vilchik Stas Vilchik (Inactive)
              • Votes:
                3 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: