Under certain circumstances, if an unexpected failure breaks the service, users need to proceed to a recovery, starting again the service from a Database backup. In this case, SonarQube restarts from fresh Elasticsearch indexes to ensure they are up to date with the restored database.
Sonarqube takes a long time to startup, exclusively because of this re-indexing.
We want SonarQube to be up and running as fast as possible, specifically for our users who decided to operate a Data Center Edition in order to benefit from a highly available service.
For this, we want to restore SonarQube availability without waiting for the whole indexing. We want to start SonarQube without initially indexing issues, which correspond generally to the biggest index in SonarQube, but index them while SonarQube is already running.
We can expect it to benefit to the upgrade of SonarQube when issues need to be re-indexed (even if this is not the main purpose of this MMF).
As a developer
- (asynchronous indexing) In case data is restored from a database snapshot, I expect SonarQube to start as fast as possible so that I can keep doing my main activities:
- access to the projects I'm working on
- see the status of the branches and PRs I'm working on
- analyze new commits and get the status updated on branches / PRs
- be able to merge based on the updated status
- not have partial information which would be confusing (ex: incomplete list of issues in my PR)
- (user feedback) While SonarQube is indexing the data, if the information I'm interested in is not available or partially available, I expect to clearly be informed that:
- the data is incomplete
- an update is in progress which will end soon, that it's a temporary situation
- (background process) I expect to be able to be able to see the information I need as soon as it's available:
- as soon as a new analysis is processed by SonarQube
- as soon as a project / branch / PR has been indexed __
- (reentrance) In case SonarQube has to be restarted again during the indexing process, I expect the update to continue and not to restart from scratch: I don't want the info for my projects to be unavailable again.
As a system administrator
- (user feedback) I expect to be able to see when the indexing is still in-progress so that I'm conscious that the service is impacted, and when it's completed.
Ideally, I expect to know where we are in the process (time remaining).
- I expect to see a background task about the issues indexing, in order to know how much time it took to index a project, and also to see on which project an issues indexing has failed
System status during the indexation process
The warning box (orange) remains visible across all pages. For a user who was on SonarQube, the warning box turns to a success box (green) when all projects have been restored. It disappears when the user dismisses it or automatically after 24 hours. (see mockups below)
Project lists during the indexation process
- Projects become available when all their issues/branches/PR have been fully restored.
- A project that becomes available turns from that grey inactive state to its normal state (colors/active link/measures visible) automatically without the page refreshing.
The visual improvements of the project box are a NICE TO HAVE. (ticket attached to the MMF)
Edit: Main project measures remain available on inactive projects (same as for the quality gate info) since the facets will keep working.
Issues page and Portfolios page temporarily unavailable
- The Issues page and Portfolios page remain unavailable until the restoration of all issues/projects is complete. The following messages will be displayed when the user tries to access one of those pages.
- Those pages refresh automatically to their normal state when issues/projects/portfolios have been fully indexed.
- A project remains unavailable until its issues/branches/Pull requests have been fully reindexed.
- The user ends up on the project overview with the following message (screenshot) as long as the project is unavailable if he/she tries to access the project from an external link (email/notification). The other pages of the project are inaccessible.
- The page refreshes automatically when the project becomes available.
Edit: Project info and project settings remain available but branch list is inactive until the project is available.
- Display issues indexing tasks in the Background Task page
- Project pages should be unavailable as long as issue indexing is in progress
- It should be explicit in Projects page which project is indexed or not
- Issues page should be unavailable as long as all projects indexing are in progress
- Portfolio pages are unavailable as long as all projects indexing are in progress
- Portfolios page is unavailable as long as all projects indexing are in progress
- Application pages are unavailable as long as all projects indexing are in progress
- If issues index is empty, or if issues indexing was started during a previous startup but not finished, start the asynchronous index indexing
- Stop computation of Portfolios/Applications if asynchronous index indexing is started
- In order to display the warning message about issues indexing, a web service should be created/update (api/navigation/global ?) to returned the fact that issues indexing is running or not
- In api/system/info, the number of projects to be indexed and the total number of projects should be returned
- Web services using issues index should fail with 503 HTTP error with message details (indexation in progress)
- If the project indexing task fails, the background task should be in error and the error should be displayed in the project page
- At startup when indexation conditions are met, 1 new task should be created for each project
- Workers will be handling tasks in a round robin mode by tasks types. That means load will be spited evenly, but giving the fact that indexation of project’s issues should be short, most of the time will be dedicated to analysis
- Indexation will be considered as finished when all tasks has been completed
(See POC from Seb for reference, this implementation differs from the POC)