Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-10479

SettingsLoader internal state is not reset and blocks CE worker

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.1
    • Component/s: Compute Engine
    • Labels:

      Description

      Under some OOM conditions during the processing of an analysis report in the Compute Engine, the internal state of the SettingsLoader class is not reset and any subsequent analysis report processing on this worker will fail with the error below:

      2018.03.09 16:20:54 ERROR ce[AWILV4fTvD4nwDDaI2ra][o.s.c.t.CeWorkerImpl] Failed to execute task AWILV4fTvD4nwDDaI2ra
      java.lang.IllegalStateException: load called twice for thread 'Worker 0 (UUID=AWHW85_zyxe3A58CkrBW) on ce-worker-0' or state wasn't cleared last time it was used
      	at com.google.common.base.Preconditions.checkState(Preconditions.java:197)
      	at org.sonar.server.setting.ThreadLocalSettings.load(ThreadLocalSettings.java:153)
      	at org.sonar.ce.settings.SettingsLoader.start(SettingsLoader.java:43)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.invokeMethod(ReflectionLifecycleStrategy.java:110)
      	at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.start(ReflectionLifecycleStrategy.java:89)
      	at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84)
      	at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169)
      	at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132)
      	at org.picocontainer.behaviors.Stored.start(Stored.java:110)
      	at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:1016)
      	at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:1009)
      	at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:767)
      	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:134)
      	at org.sonar.server.computation.task.container.TaskContainerImpl.bootup(TaskContainerImpl.java:77)
      	at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:71)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:134)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:97)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.withCustomizedThreadName(CeWorkerImpl.java:81)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:73)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:43)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      2018.03.09 16:20:54 INFO  ce[AWILV4fTvD4nwDDaI2ra][o.s.c.t.CeWorkerImpl] Executed task | project=com.rox:EmuRox | type=REPORT | id=AWILV4fTvD4nwDDaI2ra | submitter=rossdrew@github | status=FAILED | time=28ms
      

      The failure of cleanup of the tasks Pico container is indicated by log happening a bit earlier:

      2018.03.09 16:20:47 ERROR ce[AWILUBRR-T_TB3XqpAXm][o.s.s.c.t.c.TaskContainerImpl] Cleanup of container failed
      

      The tasks which failed with OOM reported the following error (see INFRA-2020 for details):

      2018.03.09 16:20:47 ERROR ce[AWILUBRR-T_TB3XqpAXm][o.s.c.t.CeWorkerImpl] Failed to execute task AWILUBRR-T_TB3XqpAXm
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      	at java.util.Arrays.copyOf(Arrays.java:3332)
      	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
      	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
      	at java.lang.StringBuilder.append(StringBuilder.java:136)
      	at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.appendDuplication(DuplicationDataMeasuresStep.java:132)
      	at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.appendDuplication(DuplicationDataMeasuresStep.java:125)
      	at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.processDuplicationBlock(DuplicationDataMeasuresStep.java:114)
      	at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.createXmlDuplications(DuplicationDataMeasuresStep.java:100)
      	at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.computeDuplications(DuplicationDataMeasuresStep.java:85)
      	at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.visitFile(DuplicationDataMeasuresStep.java:80)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:79)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:51)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:98)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:98)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
      	at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
      	at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep.execute(DuplicationDataMeasuresStep.java:67)
      	at org.sonar.server.computation.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64)
      	at org.sonar.server.computation.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52)
      	at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:73)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:134)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:97)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl$$Lambda$785/692919658.get(Unknown Source)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.withCustomizedThreadName(CeWorkerImpl.java:81)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:73)
      	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:43)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      2018.03.09 16:20:54 INFO  ce[AWILUBRR-T_TB3XqpAXm][o.s.c.t.CeWorkerImpl] Executed task | project=ll | type=REPORT | id=AWILUBRR-T_TB3XqpAXm | submitter=marium027@bitbucket | status=FAILED | time=597989ms
      

        Attachments

          Activity

            People

            Assignee:
            simon.brandhof Simon Brandhof (Inactive)
            Reporter:
            sebastien.lesaint Sebastien Lesaint
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Due:
              Created:
              Updated:
              Resolved: