Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-8075

Inconsistent search of issues on machines with more than 12 CPU cores

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.6
    • Fix Version/s: 5.6.2, 6.1
    • Component/s: ElasticSearch
    • Labels:

      Description

      Problem

      Project Homepage and Issues Page display different values. This is due to incomplete indexing of issues during server startup.

      EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1@40f3e9ce
      

      This error is a corner-case related to number of CPU cores. Server sends too many concurrent indexing requests, Elastisearch thread pool and queue do not support such load. This behaviour occurs when having more than 12 CPU cores. Probability of inconsistent data increases with the number of cores greater than 12.

      Explanation

      Thread pool and queue handle the bulk requests per shard (see "Tip #9: Threadpool Properties Prevent Data Loss"). For this reason loading capacity is (#threads + queue_size) / max nb of different shards involved in a bulk request.

      In our case, most of indices have 5 shards and default ES settings are applied (thread pool for bulks is nb of cores, queue size for bulks is 50) , so capacity of ES node is (#cores + 50) / 5 concurrent bulk requests.

      Cores Capacity
      1 10
      2 10
      12 12
      48 14
      96 29

      The problem is that the indexer client sends (#cores -1) concurrent bulk requests at server startup (see BulkIndexer.java). There are no good reasons for that.

      The consequence is that ES is overloaded and can't process all these concurrent requests:

      Cores Capacity Loading (should be <= capacity)
      1 10 1
      2 10 1
      12 12 11
      48 14 47
      96 29 95

      Solution

      Restrict indexer client to (#cores / #maxShards), so (#cores/5) concurrent bulk requests :

      Cores Capacity Loading (should be <= capacity)
      1 10 1
      2 10 1
      12 12 2
      48 14 9
      96 29 19

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              simon.brandhof Simon Brandhof (Inactive)
              Reporter:
              simon.brandhof Simon Brandhof (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Due:
                Created:
                Updated:
                Resolved: