Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-11745

NPE when migrating multi-module project

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 7.6
    • Fix Version/s: 7.8
    • Component/s: Compute Engine
    • Labels:
      None
    • Edition:
      Community
    • Production Notes:
      None

      Description

      WHY

      Failed to execute task AWkNhD5ZYuXlB8Gz59tn
      org.sonar.ce.task.projectanalysis.component.VisitException: Visit of Component {key=org.ops4j.pax:logging,type=PROJECT} failed
      	at org.sonar.ce.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
      	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:74)
      	at org.sonar.ce.task.projectanalysis.step.ExecuteVisitorsStep.execute(ExecuteVisitorsStep.java:51)
      	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
      	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
      	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
      	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:207)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:189)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:156)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:131)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:83)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:51)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.IllegalStateException: Fail to process issues of component 'org.ops4j.pax:logging'
      	at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:71)
      	at org.sonar.ce.task.projectanalysis.component.TypeAwareVisitorWrapper.visitAny(TypeAwareVisitorWrapper.java:77)
      	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitNode(VisitorsCrawler.java:117)
      	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:100)
      	at org.sonar.ce.task.projectanalysis.component.VisitorsCrawler.visit(VisitorsCrawler.java:72)
      	... 19 common frames omitted
      Caused by: java.lang.NullPointerException: null
      	at org.sonar.ce.task.projectanalysis.issue.ProjectTrackerBaseLazyInput.buildModuleProjectRelativePath(ProjectTrackerBaseLazyInput.java:99)
      	at org.sonar.ce.task.projectanalysis.issue.ProjectTrackerBaseLazyInput.buildDirectoryProjectRelativePath(ProjectTrackerBaseLazyInput.java:91)
      	at org.sonar.ce.task.projectanalysis.issue.ProjectTrackerBaseLazyInput.lambda$loadIssues$0(ProjectTrackerBaseLazyInput.java:80)
      	at java.util.ArrayList.forEach(ArrayList.java:1257)
      	at org.sonar.ce.task.projectanalysis.issue.ProjectTrackerBaseLazyInput.loadIssues(ProjectTrackerBaseLazyInput.java:77)
      	at org.sonar.core.issue.tracking.LazyInput.getIssues(LazyInput.java:50)
      	at org.sonar.core.issue.tracking.FilteringBaseInputWrapper.<init>(FilteringBaseInputWrapper.java:34)
      	at org.sonar.core.issue.tracking.NonClosedTracking.of(NonClosedTracking.java:35)
      	at org.sonar.core.issue.tracking.Tracker.trackNonClosed(Tracker.java:34)
      	at org.sonar.ce.task.projectanalysis.issue.TrackerExecution.track(TrackerExecution.java:56)
      	at org.sonar.ce.task.projectanalysis.issue.IssueTrackingDelegator.track(IssueTrackingDelegator.java:53)
      	at org.sonar.ce.task.projectanalysis.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:64)
      	... 23 common frames omitted
      

      WHAT

      The problem happen because peach DB is in an invalid state. Some issues on disabled components are marked as closed in the ES index, but not in DB.

      DB is "corrupted" only for the org.ops4j.pax:logging project. We should try to find the root cause.

      Timeline:

      • Peach was on SQ 6.7.3
      • ~04/09/2018 New release of Maven scanner with MSONAR-167 -> all modules keys are changed on pax, and issues should have been relocated
      • 25/10/2018 first evidence of creation of duplicated issues in DB, attached to removed components
      • 14/11/2018 Peach updated to SQ 7.4 (INFRA-2417)
      • 10/12/2018 all broken issues starts being duplicated in DB every day (=every analysis)
      • 03/12/2018 Peach updated to SQ 7.5.0.19649 (INFRA-2487) to fix SONAR-11529 (caused by SONAR-11394?)
      • 13/02/2019 Peach updated to SQ 7.6 (INFRA-2582)
      • no more duplication of issues in DB

      Discussion on 2019-06-05 between Julien Henry and Sebastien Lesaint concluded that the best hypothesis to explain how the data corruption happened is the following:

      1. closing issues related to disabled components happens during processing of an analysis report in the following fashion
        1. during computations steps of the analysis report processing a list of the components disabled by this analysis is built up in memory
        2. during the component persistence step of the analysis report, any component in this list is updated in DB as being disabled
        3. during the purge step of the analysis report processing, file sources and live measures of those components are deleted and any issues related to those components which is not already closed is closed
      2. if for any reason the report processing is interrupted between step 2 and step 3, the components disabled during this analysis report processing will be left with open issues for ever. 

      Our hypothesis is that the cause of the DB corruption is the fact that purge of child tables of disabled components is NOT resilient (resilient="state will be eventually recovered").

      Note 1: that this hypothesis implies that we also have "bad" data in tables FILE_SOURCES and LIVE_MEASURES but this luckily has no bad impact except disk usage so far.

      Note 2: the situation were issues are correctly indicated as closed in ES but are not in the DB can be explained by a the following code snippet]:

       

      private void purgeDisabledComponents(DbSession session, PurgeMapper mapper, PurgeConfiguration conf, PurgeListener listener) {
        executeLargeInputs(conf.getDisabledComponentUuids(),
          input -> {
            mapper.deleteFileSourcesByFileUuid(input);
            mapper.resolveComponentIssuesNotAlreadyResolved(input, system2.now());
            mapper.deleteLiveMeasuresByComponentUuids(input);
            return emptyList();
          });
      
        listener.onComponentsDisabling(conf.rootUuid(), conf.getDisabledComponentUuids()); // ES indexation
      
        session.commit();
      }
      

      If a a failures occurs while ES indexation is ongoing, then some issues will have been indexed as closed but they will never be stored as so in DB.

       

      HOW

      Fixing this bug therefor requires three operations:

      1. make purge of child table of disabled components resilient
        1. if failure occurs while or before child tables are purged, the next analysis should recover it
        2. if failure occurs while or before indexing issues closed for the disabled components, it should be recovered eventually
      2. disabled components older than some threshold are deleted (seems covered by SONAR-11585)
      3. fix the NPE above by ignoring issues of disabled components

      *1 will fix the DB corruption

      *2 will prevent the number of disabled components for a given project to be an ever growing factor and prevent *1 to become a forever increasing costly operation

      *3 will allow the report processing of a corrupted project to actually reach the new purge code which will remove the DB corruption.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sebastien.lesaint Sebastien Lesaint
                Reporter:
                julien.henry Julien Henry
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: