Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-9286

Elasticsearch error during startup in DataCenter Edition

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.4
    • Component/s: None
    • Labels:

      Description

      During first startup, there is the following error :

      > 2017.05.18 16:46:56 INFO  es[][o.e.index.shard] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [users][0] updating refresh_interval from [-1] to [30s]
      > 2017.05.18 16:46:56 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] closing indices [[users]]
      > 2017.05.18 16:46:56 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] opening indices [[users]]
      > 2017.05.18 16:46:56 INFO  es[][o.e.c.r.allocation] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[users][0]] ...]).
      > 2017.05.18 16:46:56 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [users/user] done | time=631ms
      > 2017.05.18 16:46:56 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [components/component] ...
      > 2017.05.18 16:46:56 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] updating number_of_replicas to [0] for indices [components]
      > 2017.05.18 16:46:56 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] updating number_of_replicas to [1] for indices [components]
      > 2017.05.18 16:46:56 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] closing indices [[components]]
      > 2017.05.18 16:46:57 WARN  es[][o.e.indices.cluster] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [[components][0]] marking and sending shard failed due to [failed recovery]
      > org.elasticsearch.indices.recovery.RecoveryFailedException: [components][0]: Recovery failed from {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true} into {sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b}{9OKkBO1KRneKq_voqcFBdg}{192.168.1.30}{192.168.1.30:33083}{rack_id=sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b, master=true}
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:258)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.access$1100(RecoveryTarget.java:69)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:508)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.transport.RemoteTransportException: [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090][192.168.1.30:41757][internal:index/shard/recovery/start_recovery]
      > Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:126)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:52)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
      > 	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
      > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
      > 	at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [1] files with total size of [130b]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:453)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:133)
      > 	... 11 common frames omitted
      > Caused by: org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: recovery was canceled reason [shard is closed]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler$1.onCancel(RecoverySourceHandler.java:93)
      > 	at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:104)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:375)
      > 	... 12 common frames omitted
      > 2017.05.18 16:46:57 WARN  es[][o.e.c.action.shard] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [components][0] received shard failed for target shard [[components][0], node[9OKkBO1KRneKq_voqcFBdg], [R], v[4], s[INITIALIZING], a[id=4iJeZBYkRbiRT_8xSK6ogA], unassigned_info[[reason=REPLICA_ADDED], at[2017-05-18T14:46:56.883Z]]], indexUUID [dBa1KaIEQuiycuE0AQU14g], message [failed recovery], failure [RecoveryFailedException[[components][0]: Recovery failed from {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true} into {sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b}{9OKkBO1KRneKq_voqcFBdg}{192.168.1.30}{192.168.1.30:33083}{rack_id=sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b, master=true}]; nested: RemoteTransportException[[sonarqube-fa49f042-e142-445a-9d7b-672dad07c090][192.168.1.30:41757][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [130b]]; nested: ExecutionCancelledException[recovery was canceled reason [shard is closed]]; ]
      > org.elasticsearch.indices.recovery.RecoveryFailedException: [components][0]: Recovery failed from {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true} into {sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b}{9OKkBO1KRneKq_voqcFBdg}{192.168.1.30}{192.168.1.30:33083}{rack_id=sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b, master=true}
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:258)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.access$1100(RecoveryTarget.java:69)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:508)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.transport.RemoteTransportException: [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090][192.168.1.30:41757][internal:index/shard/recovery/start_recovery]
      > Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:126)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:52)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
      > 	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
      > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
      > 	at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [1] files with total size of [130b]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:453)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:133)
      > 	... 11 common frames omitted
      > Caused by: org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: recovery was canceled reason [shard is closed]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler$1.onCancel(RecoverySourceHandler.java:93)
      > 	at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:104)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:375)
      > 	... 12 common frames omitted
      > 2017.05.18 16:46:57 WARN  es[][o.e.indices.cluster] [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090] [[components][4]] marking and sending shard failed due to [failed recovery]
      > org.elasticsearch.indices.recovery.RecoveryFailedException: [components][4]: Recovery failed from {sonarqube-428170e0-8718-4d05-9b1e-db653da148ee}{x-6Aa7q8SEqX1VF61xI7zA}{192.168.1.30}{192.168.1.30:35949}{rack_id=sonarqube-428170e0-8718-4d05-9b1e-db653da148ee, master=true} into {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true}
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:258)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.access$1100(RecoveryTarget.java:69)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:508)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.transport.RemoteTransportException: [sonarqube-428170e0-8718-4d05-9b1e-db653da148ee][192.168.1.30:35949][internal:index/shard/recovery/start_recovery]
      > Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:126)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:52)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
      > 	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
      > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
      > 	at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [1] files with total size of [130b]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:453)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:133)
      > 	... 11 common frames omitted
      > Caused by: org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: recovery was canceled reason [shard is closed]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler$1.onCancel(RecoverySourceHandler.java:93)
      > 	at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:104)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.prepareTargetForTranslog(RecoverySourceHandler.java:463)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:448)
      > 	... 12 common frames omitted
      > 	Suppressed: java.lang.IllegalStateException: Future got interrupted
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47)
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:33)
      > 		at org.elasticsearch.indices.recovery.RecoverySourceHandler$6.run(RecoverySourceHandler.java:469)
      > 		at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86)
      > 		... 14 common frames omitted
      > 	Caused by: java.lang.InterruptedException: null
      > 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      > 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      > 		at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:280)
      > 		at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:120)
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44)
      > 		... 17 common frames omitted
      > 2017.05.18 16:46:57 WARN  es[][o.e.c.action.shard] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [components][4] received shard failed for target shard [[components][4], node[rUVla4X7Q_uTRy9cxpyRgA], [R], v[5], s[INITIALIZING], a[id=0_5MtlLrTLKWqhwOdUW7Hg], unassigned_info[[reason=REPLICA_ADDED], at[2017-05-18T14:46:56.883Z]]], indexUUID [dBa1KaIEQuiycuE0AQU14g], message [failed recovery], failure [RecoveryFailedException[[components][4]: Recovery failed from {sonarqube-428170e0-8718-4d05-9b1e-db653da148ee}{x-6Aa7q8SEqX1VF61xI7zA}{192.168.1.30}{192.168.1.30:35949}{rack_id=sonarqube-428170e0-8718-4d05-9b1e-db653da148ee, master=true} into {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true}]; nested: RemoteTransportException[[sonarqube-428170e0-8718-4d05-9b1e-db653da148ee][192.168.1.30:35949][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [130b]]; nested: ExecutionCancelledException[recovery was canceled reason [shard is closed]]; ]
      > org.elasticsearch.indices.recovery.RecoveryFailedException: [components][4]: Recovery failed from {sonarqube-428170e0-8718-4d05-9b1e-db653da148ee}{x-6Aa7q8SEqX1VF61xI7zA}{192.168.1.30}{192.168.1.30:35949}{rack_id=sonarqube-428170e0-8718-4d05-9b1e-db653da148ee, master=true} into {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true}
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:258)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.access$1100(RecoveryTarget.java:69)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:508)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.transport.RemoteTransportException: [sonarqube-428170e0-8718-4d05-9b1e-db653da148ee][192.168.1.30:35949][internal:index/shard/recovery/start_recovery]
      > Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:126)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:52)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
      > 	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
      > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
      > 	at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [1] files with total size of [130b]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:453)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:133)
      > 	... 11 common frames omitted
      > Caused by: org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: recovery was canceled reason [shard is closed]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler$1.onCancel(RecoverySourceHandler.java:93)
      > 	at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:104)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.prepareTargetForTranslog(RecoverySourceHandler.java:463)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:448)
      > 	... 12 common frames omitted
      > 	Suppressed: java.lang.IllegalStateException: Future got interrupted
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47)
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:33)
      > 		at org.elasticsearch.indices.recovery.RecoverySourceHandler$6.run(RecoverySourceHandler.java:469)
      > 		at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86)
      > 		... 14 common frames omitted
      > 	Caused by: java.lang.InterruptedException: null
      > 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      > 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      > 		at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:280)
      > 		at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:120)
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44)
      > 		... 17 common frames omitted
      > 2017.05.18 16:46:57 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] opening indices [[components]]
      > 2017.05.18 16:46:57 INFO  es[][o.e.c.r.allocation] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[components][2], [components][2]] ...]).
      > 2017.05.18 16:46:57 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [components/component] done | time=565ms
      > 2017.05.18 16:46:57 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [views/view] ...
      > 2017.05.18 16:46:57 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] updating number_of_replicas to [0] for indices [views]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090] [views][1] updating refresh_interval from [30s] to [-1]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090] [views][4] updating refresh_interval from [30s] to [-1]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-428170e0-8718-4d05-9b1e-db653da148ee] [views][2] updating refresh_interval from [30s] to [-1]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [views][3] updating refresh_interval from [30s] to [-1]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [views][0] updating refresh_interval from [30s] to [-1]
      > 2017.05.18 16:46:57 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] updating number_of_replicas to [1] for indices [views]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090] [views][1] updating refresh_interval from [-1] to [30s]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-428170e0-8718-4d05-9b1e-db653da148ee] [views][2] updating refresh_interval from [-1] to [30s]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090] [views][4] updating refresh_interval from [-1] to [30s]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [views][3] updating refresh_interval from [-1] to [30s]
      > 2017.05.18 16:46:57 INFO  es[][o.e.index.shard] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [views][0] updating refresh_interval from [-1] to [30s]
      > 2017.05.18 16:46:57 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] closing indices [[views]]
      > 2017.05.18 16:46:57 WARN  es[][o.e.indices.cluster] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [[views][1]] marking and sending shard failed due to [failed recovery]
      > org.elasticsearch.indices.recovery.RecoveryFailedException: [views][1]: Recovery failed from {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true} into {sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b}{9OKkBO1KRneKq_voqcFBdg}{192.168.1.30}{192.168.1.30:33083}{rack_id=sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b, master=true}
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:258)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.access$1100(RecoveryTarget.java:69)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:508)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.transport.RemoteTransportException: [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090][192.168.1.30:41757][internal:index/shard/recovery/start_recovery]
      > Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:126)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:52)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
      > 	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
      > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
      > 	at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [1] files with total size of [130b]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:453)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:133)
      > 	... 11 common frames omitted
      > Caused by: org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: recovery was canceled reason [shard is closed]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler$1.onCancel(RecoverySourceHandler.java:93)
      > 	at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:104)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.prepareTargetForTranslog(RecoverySourceHandler.java:463)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:448)
      > 	... 12 common frames omitted
      > 	Suppressed: java.lang.IllegalStateException: Future got interrupted
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47)
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:33)
      > 		at org.elasticsearch.indices.recovery.RecoverySourceHandler$6.run(RecoverySourceHandler.java:469)
      > 		at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86)
      > 		... 14 common frames omitted
      > 	Caused by: java.lang.InterruptedException: null
      > 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      > 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      > 		at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:280)
      > 		at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:120)
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44)
      > 		... 17 common frames omitted
      > 2017.05.18 16:46:57 WARN  es[][o.e.c.action.shard] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] [views][1] received shard failed for target shard [[views][1], node[9OKkBO1KRneKq_voqcFBdg], [R], v[5], s[INITIALIZING], a[id=ba9-JTAeSim6qwkZJ_nJpw], unassigned_info[[reason=REPLICA_ADDED], at[2017-05-18T14:46:57.583Z]]], indexUUID [1eV6KK6tQLWIOQUv58ifvg], message [failed recovery], failure [RecoveryFailedException[[views][1]: Recovery failed from {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true} into {sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b}{9OKkBO1KRneKq_voqcFBdg}{192.168.1.30}{192.168.1.30:33083}{rack_id=sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b, master=true}]; nested: RemoteTransportException[[sonarqube-fa49f042-e142-445a-9d7b-672dad07c090][192.168.1.30:41757][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [130b]]; nested: ExecutionCancelledException[recovery was canceled reason [shard is closed]]; ]
      > org.elasticsearch.indices.recovery.RecoveryFailedException: [views][1]: Recovery failed from {sonarqube-fa49f042-e142-445a-9d7b-672dad07c090}{rUVla4X7Q_uTRy9cxpyRgA}{192.168.1.30}{192.168.1.30:41757}{rack_id=sonarqube-fa49f042-e142-445a-9d7b-672dad07c090, master=true} into {sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b}{9OKkBO1KRneKq_voqcFBdg}{192.168.1.30}{192.168.1.30:33083}{rack_id=sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b, master=true}
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:258)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget.access$1100(RecoveryTarget.java:69)
      > 	at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:508)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.transport.RemoteTransportException: [sonarqube-fa49f042-e142-445a-9d7b-672dad07c090][192.168.1.30:41757][internal:index/shard/recovery/start_recovery]
      > Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:126)
      > 	at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:52)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:135)
      > 	at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
      > 	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
      > 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
      > 	at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:293)
      > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
      > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      > 	at java.lang.Thread.run(Thread.java:748)
      > Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [1] files with total size of [130b]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:453)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:133)
      > 	... 11 common frames omitted
      > Caused by: org.elasticsearch.common.util.CancellableThreads$ExecutionCancelledException: recovery was canceled reason [shard is closed]
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler$1.onCancel(RecoverySourceHandler.java:93)
      > 	at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:104)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.prepareTargetForTranslog(RecoverySourceHandler.java:463)
      > 	at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:448)
      > 	... 12 common frames omitted
      > 	Suppressed: java.lang.IllegalStateException: Future got interrupted
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:47)
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:33)
      > 		at org.elasticsearch.indices.recovery.RecoverySourceHandler$6.run(RecoverySourceHandler.java:469)
      > 		at org.elasticsearch.common.util.CancellableThreads.execute(CancellableThreads.java:86)
      > 		... 14 common frames omitted
      > 	Caused by: java.lang.InterruptedException: null
      > 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
      > 		at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      > 		at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:280)
      > 		at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:120)
      > 		at org.elasticsearch.transport.PlainTransportFuture.txGet(PlainTransportFuture.java:44)
      > 		... 17 common frames omitted
      > 2017.05.18 16:46:57 INFO  es[][o.e.cluster.metadata] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] opening indices [[views]]
      > 2017.05.18 16:46:57 INFO  es[][o.e.c.r.allocation] [sonarqube-d4b18df6-8d07-4777-885f-cf38a586772b] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[views][4], [components][2], [views][4], [views][0], [views][3]] ...]).
      > 2017.05.18 16:46:58 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [views/view] done | time=690ms
      > 2017.05.18 16:46:58 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [issues/issue] ...
      

        Attachments

          Activity

            People

            • Assignee:
              eric.hartmann Eric Hartmann
              Reporter:
              eric.hartmann Eric Hartmann
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: