Uploaded image for project: 'Product Roadmaps'
  1. Product Roadmaps
  2. MMF-1671

Data migration from 6.7 LTS to 7.9 LTS should require a 'reasonable' amount of time with all DBs

    XMLWordPrintable

    Details

    • Type: MMF
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Labels:

      Description

      WHY

      Most of big SonarQube users rely on LTS versions and are doing a LTS-to-LTS upgrade once a year. This upgrade can take a big amount of time mainly due to all the updates that should be applied on the DB structure and due to regeneration of the ES indexes.

      WHAT

      With a SonarQube instance containing 100M issues and about 5'000 projects, we don't expect the full upgrade to take more than 24 hours on a dedicated infrastructure to be defined and with all the supported DB:

      • SQL Server
      • Oracle
      • Postgresql. 

      HOW 

      Prepare the 6.7 infra

      We need a DCE edition with a license to analyze a lot of LoC. The DB should be PostgresSQL 9.3. See INFRA-1705 for the sizing of the previous attempt. The DCE edition should be configured with a lot of CE workers.
      We could get inspiration from what was done for SonarSecurity perf: BUILD-736

      Prepare the sample project in a GitHub repo

      We need a (multi-module ?) project compatible with autoscan, including test/coverage reports (pre generated and stored in the repo), and various branches
      In order to have good dataset in a 6.7, let's take a base project, size M or L, with 5 long living branches and at least 100 files analyzed. 
      YetiForceCRM is a good candidate, it's a size L project with 5 different languages already analyzed on SonarCloud: https://sonarcloud.io/dashboard?id=YetiForceCRM

      Schedule tons of analyses to feed the SQ 6.7 DB

      Create a dataset by duplicating this base project and analyze it with different project keys/branch/date. Target:

      • 5'000 projects
      • 15'000 long-living branches (3 per project)
      • 15'000 short-living branches (3 per project)
      • Currently YetiForceCRM has 7,742 issues =>  more than 100M issues total

      Idea: reuse autoscan architecture (queue + workers). We would just have to write a script that would put payloads in the queue.

      Optional: if time permit, create a script to comment/change status of random issues to create issue changelog.

       h3. Take a snapshot of the SQ 6.7 DB that will be used as a baseline

      Run migration to 7.9 LTS

      Collect migration logs and report hotspots to the dev team. Iterate if needed.

      Once all hotspots are fixed for PostgresSQL, use DBCopy to create baseline for Oracle 11g and SQLServer 2014, and confirm migration is fine.

        Attachments

        1. 1 DbMigration summary.log
          32 kB
        2. 1 indexation summary.log
          2 kB
        3. image-2019-06-11-10-24-18-711.png
          image-2019-06-11-10-24-18-711.png
          102 kB
        4. image-2019-06-24-10-09-30-508.png
          image-2019-06-24-10-09-30-508.png
          235 kB
        5. image-2019-06-24-10-32-02-072.png
          image-2019-06-24-10-32-02-072.png
          407 kB
        6. oracle-try-1.log
          39 kB
        7. try2-postgresql.log
          37 kB

          Issue Links

            Activity

              People

              Assignee:
              julien.henry Julien Henry
              Reporter:
              xavier.bourguignon Xavier Bourguignon (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: