Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 7.1
-
Component/s: Compute Engine
-
Labels:
Description
Under some OOM conditions during the processing of an analysis report in the Compute Engine, the internal state of the SettingsLoader class is not reset and any subsequent analysis report processing on this worker will fail with the error below:
2018.03.09 16:20:54 ERROR ce[AWILV4fTvD4nwDDaI2ra][o.s.c.t.CeWorkerImpl] Failed to execute task AWILV4fTvD4nwDDaI2ra java.lang.IllegalStateException: load called twice for thread 'Worker 0 (UUID=AWHW85_zyxe3A58CkrBW) on ce-worker-0' or state wasn't cleared last time it was used at com.google.common.base.Preconditions.checkState(Preconditions.java:197) at org.sonar.server.setting.ThreadLocalSettings.load(ThreadLocalSettings.java:153) at org.sonar.ce.settings.SettingsLoader.start(SettingsLoader.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.invokeMethod(ReflectionLifecycleStrategy.java:110) at org.picocontainer.lifecycle.ReflectionLifecycleStrategy.start(ReflectionLifecycleStrategy.java:89) at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84) at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169) at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132) at org.picocontainer.behaviors.Stored.start(Stored.java:110) at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:1016) at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:1009) at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:767) at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:134) at org.sonar.server.computation.task.container.TaskContainerImpl.bootup(TaskContainerImpl.java:77) at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:71) at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:134) at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:97) at org.sonar.ce.taskprocessor.CeWorkerImpl.withCustomizedThreadName(CeWorkerImpl.java:81) at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:73) at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:43) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018.03.09 16:20:54 INFO ce[AWILV4fTvD4nwDDaI2ra][o.s.c.t.CeWorkerImpl] Executed task | project=com.rox:EmuRox | type=REPORT | id=AWILV4fTvD4nwDDaI2ra | submitter=rossdrew@github | status=FAILED | time=28ms
The failure of cleanup of the tasks Pico container is indicated by log happening a bit earlier:
2018.03.09 16:20:47 ERROR ce[AWILUBRR-T_TB3XqpAXm][o.s.s.c.t.c.TaskContainerImpl] Cleanup of container failed
The tasks which failed with OOM reported the following error (see INFRA-2020 for details):
2018.03.09 16:20:47 ERROR ce[AWILUBRR-T_TB3XqpAXm][o.s.c.t.CeWorkerImpl] Failed to execute task AWILUBRR-T_TB3XqpAXm java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) at java.lang.StringBuilder.append(StringBuilder.java:136) at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.appendDuplication(DuplicationDataMeasuresStep.java:132) at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.appendDuplication(DuplicationDataMeasuresStep.java:125) at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.processDuplicationBlock(DuplicationDataMeasuresStep.java:114) at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.createXmlDuplications(DuplicationDataMeasuresStep.java:100) at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.computeDuplications(DuplicationDataMeasuresStep.java:85) at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep$DuplicationVisitor.visitFile(DuplicationDataMeasuresStep.java:80) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:79) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:51) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:98) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:98) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54) at org.sonar.server.computation.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39) at org.sonar.server.computation.task.projectanalysis.step.DuplicationDataMeasuresStep.execute(DuplicationDataMeasuresStep.java:67) at org.sonar.server.computation.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64) at org.sonar.server.computation.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52) at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:73) at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:134) at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:97) at org.sonar.ce.taskprocessor.CeWorkerImpl$$Lambda$785/692919658.get(Unknown Source) at org.sonar.ce.taskprocessor.CeWorkerImpl.withCustomizedThreadName(CeWorkerImpl.java:81) at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:73) at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:43) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) 2018.03.09 16:20:54 INFO ce[AWILUBRR-T_TB3XqpAXm][o.s.c.t.CeWorkerImpl] Executed task | project=ll | type=REPORT | id=AWILUBRR-T_TB3XqpAXm | submitter=marium027@bitbucket | status=FAILED | time=597989ms