Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-8005

WS api/issues/search fails when requested page is too high

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.0
    • Fix Version/s: 6.2
    • Component/s: ElasticSearch, Issues
    • Labels:

      Description

      Error is :

      Fail to process request ..../api/issues/search?p=101
      java.lang.IllegalStateException: Fail to execute ES search request '{"from":10000,"size":100,"query":{"bool":{"must":{"match_all":{}},"filter":{"bool":{"must":{"has_parent":{"query":{"bool":{"must":{"match_all":{}},"filter":{"bool":{"should":{"term":{"groups":"Anyone"}}}}}},"parent_type":"authorization"}}}}}},"sort":[{"issueUpdatedAt":{"order":"desc","missing":"_first"}},{"key":{"order":"asc","missing":"_first"}}]}' on indices '[issues]' on types '[issue]'
      	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:48) ~[sonar-server-6.0.jar:na]
      	at org.sonar.server.es.request.ProxySearchRequestBuilder.get(ProxySearchRequestBuilder.java:36) ~[sonar-server-6.0.jar:na]
      	at org.sonar.server.issue.index.IssueIndex.search(IssueIndex.java:237) ~[sonar-server-6.0.jar:na]
      	at org.sonar.server.issue.ws.SearchAction.doHandle(SearchAction.java:301) ~[sonar-server-6.0.jar:na]
      	at org.sonar.server.issue.ws.SearchAction.handle(SearchAction.java:288) ~[sonar-server-6.0.jar:na]
      	at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:109) ~[sonar-server-6.0.jar:na]
      	at org.sonar.server.ws.WebServiceFilter.doFilter(WebServiceFilter.java:74) [sonar-server-6.0.jar:na]
      	at org.sonar.server.platform.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:125) [sonar-server-6.0.jar:na]
      	at org.sonar.server.platform.MasterServletFilter.doFilter(MasterServletFilter.java:94) [sonar-server-6.0.jar:na]
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.sonar.server.user.UserSessionFilter.doFilter(UserSessionFilter.java:56) [sonar-server-6.0.jar:na]
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.sonar.server.platform.RoutesFilter.doFilter(RoutesFilter.java:55) [sonar-server-6.0.jar:na]
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.sonar.server.platform.ProfilingFilter.doFilter(ProfilingFilter.java:84) [sonar-server-6.0.jar:na]
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:191) [logback-access-1.1.3.jar:na]
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
      	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.0.32.jar:8.0.32]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
      Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
      	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:855) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:833) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:387) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
      	... 1 common frames omitted
      Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Result window is too large, from + size must be less than or equal to: [10000] but was [10100]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.
      	at org.elasticsearch.search.internal.DefaultSearchContext.preProcess(DefaultSearchContext.java:212) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:103) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.search.SearchService.createContext(SearchService.java:676) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376) ~[elasticsearch-2.3.3.jar:2.3.3]
      	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-2.3.3.jar:2.3.3]
      	... 3 common frames omitted
      

      Elasticsearch returns an error message if a query’s from + size is more than the "index.max_result_window parameter". This parameter defaults to 10'000 which is safe for almost all clusters. Values higher than can consume significant chunks of heap memory per search and per shard executing the search.

      The safest implementation would be to use the Elasticsearch scroll api, but instead of this technical trick the WS must return BAD REQUEST if the requested range cannot be proceed (page number * page size >= 10'000).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                teryk.bellahsene Teryk Bellahsene (Inactive)
                Reporter:
                simon.brandhof Simon Brandhof
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: