Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Do
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: ElasticSearch
    • Labels:
      None
    • Edition:
      Community
    • Production Notes:
      None

      Description

      WHY

      SonarQube is bundled with Elasticsearch 6.8 since 7.X LTS.

      Elasticsearch 7.0 was released on April 10 of 2019. SonarQube LTS was released in July 2019.

      As of today, the latest release of Elasticsearch is 7.4.2 (released on October 31, 2019).

      According to this page, Elasticsearch 6.8.X will be EOL in November 2020 and maintained until the release of ElasticSearch 8.0.0.

      Next LTS of SonarQube is planned for November 2020

      There is no way to know today whether Elasticsearch 8.X will be available and stable enough to be included in the next LTS of SonarQube, but for sure, we will have to upgrade to 7.X.

      Since with Elasticsearch most changes of a given version come with its first milestone, the sooner we upgrade to 7.X the better. It will both give us more visibility and feedback on the impacts and give us time to leverage new features to improve SonarQube.

      WHAT

      See release blog posts for overview of the changes: 7.0.0, 7.2.0, 7.4.0

      Lighter/OSS ES distribution

      Since 7.0.0 Elastic offers what they call "The pure Apache 2.0 licensed distribution". It seems this is basically Elasticsearch without any of the paid feature.

      Packaging Elasticsearch into SQ means unpacking the official zip into the sonar-enterprise repository and exclude, at packaging time, any of the files we don't use nor need .

      (this way of doing makes it explicite what we don't ship, way better than not committing the files, and makes upgrading Elasticsearch files super easy – just replace the old directory with the one from the new archive)

      We exclude all paying feature from Elastic which we exclude anyway. So, we will save on:

      1. size of the sonar-enterprise repository
      2. code to exclude the paying features

      Moving to HTTP

      HTTP port always open

      Starting with 7.0, there won't be any way to disable HTTP port opened by ElasticSearch (it's been announced for several versions).
      On the other hand, when running on a single node, there is no need to open a TCP port. TCP port is required only for ES Cluster for communication between nodes.

      On no-DCE editions:

      • port 9001 is used for HTTP communication with ElasticSearch
      • existing property "sonar.search.host" is used to change HTTP port and default value is not changed (9001)
      • property "sonar.search.httpPort" is not supported anymore
      • HTTP port is bound to localhost by default, otherwise can be changed by existing property "sonar.search.host"

      On DCE edition:

      • port 9001 will be used for HTTP communication with ElasticSearch from the APP nodes
      • same decisions as non-DCE editions for "sonar.search.host" and "sonar.search.httpPort"
      • new property "sonar.search.transportPort" is used to customize the TCP port used by ES nodes to communicate with each other
        • defaults to 9002
        • value 0 is not supported
      • HTTP and TCP ports are bound to the value of existing property "sonar.search.host"

      High-level REST client for Java

      Elastic announced that TCP Transport client (used today by SQ to query ES over TCP) is deprecated and will be dropped in ES 8.X.
      The replacement, the high-level REST client for Java, is ready with the release of 7.0.0.

      SQ should now use HTTP to query ElasticSearch.

      Side benefit: it allows to open only one port for ElasticSearch on non-DCE editions and reuse 9001, which will only make updates smoother.

      No specific action is taken to secure the HTTP port of ElasticSearch, except binding on localhost on non-DCE editions:

      • that's how TCP port is "secured" today
      • securing all SQ ports will be specific effort

      Docker images

      No impact is expected on Docker images as communication between SQ and ES happens inside the container.

      Changes in 7.X impacting SonarQube

      For migration to 6.X, all the upgrade and migration notes have been compiled into a Google Sheet to track which where irrelevant, to do, done, etc.

      The small Java program which parses the Elasticsearch Upgrade and migration notes to generate that Google sheet has been updated for 7.4. Source code is available in this GIST and a sheet from it's output if visible here.

      JSON Logging

      Since 7.0, ES logs in json files on top of text files. Since SQ defines it's own logging configuration for ES, we are not affected by this change.

      OS/Java support

      ES 7.0 drops support for Ubuntu 14.04.

       transitively, this means SonarQube can't support Ubuntu 14.04, doesn't it?

      ES 7.4 restores support for Java 11 and Java 13 (added in 6.8 but lost with 7.0, 7.1, 7.2 and 7.3), drops supports for Java 12 (provided by 7.0, 7.1, 7.2 and 7.3)

      Indices migration

      As usual, there won't be any migration of the indices. All indices from 6.8 will be dropped and new indices with 7.X will be created.

      The usual implementation of this is to rename ES data directory from es-6 to es-7.

      Note: Elastic provides documentation on the migration (see here) and indicates that, eg., migration from 6.8 to 7.0 is not supported (but 6.8 to 7.4)

      New features

      Typeahead Search

      ES 7.2 introduces native support for type ahead search. SQ provides such feature in global search (and maybe other places).

      The important point is that this native support is enabled by a new features called top-k faster queries and search_as_you_type which we may want to leverage for our own searches.

      API/Java breaking changes

      See here for more detailed info what problems should be fixed with these release.

      HOW

      Update docs that we don't support Ubuntu 14.04.

      Turn on deprecation logging on feature branch to track if something deprecated is used

      JSON Logging can be disabled by changing the log layout as seen here

      Documentation updates

      Upgrade notes:

      • DCE requires configuring a new and specific port between search nodes
      • any editions: search port is now used for HTTP traffic
      • "sonar.search.httpPort" property is now ignored

      DCE documentation

      • "sonar.search.port" is now used for communication between Application nodes and Search nodes only (not for communication between Search nodes) and will be used for HTTP traffic
      • new "sonar.search.transportPort" settings, defaults to 9002, used for TCP communication between Search nodes
      • changes to node configuration
        • "sonar.search.host" and "sonar.search.port" must be configured on each SEARCH node with the host and port the node will listen to for HTTP queries from the APP nodes
        • "sonar.cluster.search.hosts" must be
          1. configured with host and port, no more fallback onto the value of "sonar.search.host" (and anyway, it didn't work, fallback was to 9001)
          2. values are different on SEARCH and APP nodes:
            • SEARCH node: hosts and TCP ports of all SEARCH nodes
            • APP node: hosts and HTTP ports of all SEARCH nodes
      • (hardening) "sonar.cluster.hosts" should not be provided on SEARCH nodes (ignored since 7.2){{}}

       Missing chapter on Securing SonarQube and/or warning next to setting controlling SQ TCP/HTTP ports?

      • non-DCE editions: SQ binds by default to loopback address. When changing that, be very careful to secure access to these ports
      • DCE edition: we advise to configure firewall rules to only allow access from SQ nodes

       Documentation of SonarQube ports

      • network protocol used for these ports is missing (it's TCP everywhere)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jacek.poreda Jacek Poreda
                Reporter:
                sebastien.lesaint Sebastien Lesaint
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: