Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-10116

Better scalability of loading of project measures

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0
    • Component/s: Database, Measures
    • Labels:

      Description

      Loading measures from database can be quite slow when many projects have been analyzed or when projects have a long history.

      Current state

      History of all analyses of only Projects is stored in 'project_measures'
      Data of last analysis of all Components is stored in 'project_measures'
      Historical data of all Components gets purged from 'project_measures'

      Future structure

      Past measures of all components except files, and optionally directories are stored in project_measures
      "Last" measures, as computed by analyses of all components except files, and optionally directories are stored in project_measures
      Live measures of all components are in live_measures

      What needs to be done?

      • Create table live_measures
      • Migrate all data from project_measures, that relate to a latest analysis of a project
      • Let the compute engine store only project level data into project_measures, and all levels into live_measures
      • Make web services read from live_measures instead of project_measures. Some examples:
        • All issue update web services (when generating the response)
        • api/measures/component
        • api/measures/component_tree
        • api/measures/search
        • api/qualitygate/project_status
        • api/branches/list
        • api/duplications/show
        • api/ui/component
      • Update purge mechanisms:
        • Deletion of non-project level data, that is not related to a last analysis
        • Deletion of metrics (if such mechanism exists)
        • Deletion of projects
        • Deletion of organization
      • Update the governance plugin:
        • import of project dump
        • computation of applications and portfolios
      • Update the branch plugin

      After everything is updated, a last step will be:

      • Update the new database migration step, to also delete all non-project related data from project_measures

      Elasticsearch indexing has not to be changed right now, because only project level data is indexed (and that data will remain in both tables)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                daniel.schwarz Daniel Schwarz (Inactive)
                Reporter:
                daniel.schwarz Daniel Schwarz (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: