Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-10116

Better scalability of loading of project measures



    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0
    • Component/s: Database, Measures
    • Labels:


      Loading measures from database can be quite slow when many projects have been analyzed or when projects have a long history.

      Current state

      History of all analyses of only Projects is stored in 'project_measures'
      Data of last analysis of all Components is stored in 'project_measures'
      Historical data of all Components gets purged from 'project_measures'

      Future structure

      Past measures of all components except files, and optionally directories are stored in project_measures
      "Last" measures, as computed by analyses of all components except files, and optionally directories are stored in project_measures
      Live measures of all components are in live_measures

      What needs to be done?

      • Create table live_measures
      • Migrate all data from project_measures, that relate to a latest analysis of a project
      • Let the compute engine store only project level data into project_measures, and all levels into live_measures
      • Make web services read from live_measures instead of project_measures. Some examples:
        • All issue update web services (when generating the response)
        • api/measures/component
        • api/measures/component_tree
        • api/measures/search
        • api/qualitygate/project_status
        • api/branches/list
        • api/duplications/show
        • api/ui/component
      • Update purge mechanisms:
        • Deletion of non-project level data, that is not related to a last analysis
        • Deletion of metrics (if such mechanism exists)
        • Deletion of projects
        • Deletion of organization
      • Update the governance plugin:
        • import of project dump
        • computation of applications and portfolios
      • Update the branch plugin

      After everything is updated, a last step will be:

      • Update the new database migration step, to also delete all non-project related data from project_measures

      Elasticsearch indexing has not to be changed right now, because only project level data is indexed (and that data will remain in both tables)


          Issue Links



              daniel.schwarz Daniel Schwarz (Inactive)
              daniel.schwarz Daniel Schwarz (Inactive)
              0 Vote for this issue
              3 Start watching this issue