Uploaded image for project: 'Product Roadmaps'
  1. Product Roadmaps
  2. MMF-1038

Ensure Elasticsearch-Database consistency on project/portfolio purge

    XMLWordPrintable

    Details

    • Type: MMF
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Labels:

      Description

      Context

      Project/portfolio computation
      If the computation of a project/portfolio analysis is interrupted while ES indexation is ongoing, the next successful computation of this project/portfolio will update again in ES all the related data and consistency will be automatically restored.
      Indeed, as for a project analysis, each time the Compute Engine deals with the computation of a portfolio, all the related data found in DB are indexed:

      • views are updated by comparing the content of ES with the DB,
      • components, measures and issues are updated by indexing the whole portfolio.

      Purge
      Also, at the end of each project/portfolio analysis, a purge is triggered and some old data are clean out. The purge updates components and issues, deleting the data of past analyses from both the DB and ES. Changes are effectively committed in DB after ES has been updated.
      As a consequence, if the purge is interrupted, the purge to be triggered by the next analysis will find again the data in DB and will delete it from ES.

      That works when the task / the indexation is interrupted. But, to be able to rely on this mechanism when an error happens while trying to update data in Elasticsearch, we need to make to sure that we catch the errors that can occur so that the task is interrupted.

      Solution

      Project/portfolio computation
      If a failure happens while indexing the data related to a project/portfolio, the error should be catched, indexation should be interrupted and the task should fail so that the user knows that project/portfolio has to be analyzed again.

      Purge
      If a failure happens during the purge while cleaning up ES, the error should be catched and the purge should be interrupted, the task should fail before changes are committed in DB so that the next analysis can purge again the data.

      Related changes

      Full reindexation at server startup
      In some cases (upgrade, corruption of data), the content of ES may need to be deleted and fully restored from the DB.
      As soon as an indexation request fails, the startup should fail so that the next time the server will start again, the server will drop and restore again the indexes for which indexation was not completed.

      Project import
      We should make sure that, if there's a failure during the import of a project, the import is considered as failed so that the user knows he has to import again the project.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              christophe.levis Christophe Levis
              Reporter:
              christophe.levis Christophe Levis
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: