Uploaded image for project: 'SonarQube'
  1. SonarQube
  2. SONAR-8995

Scanner fails when using issue exclusions

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.3
    • Fix Version/s: 6.3.1, 6.4
    • Component/s: Scanner
    • Labels:

      Description

      Problem

      Certain issue filters cause SonarQube to read all indexed files before running the sensors.
      Any settings that involves excluding files or rules based on the content of the files matching some regular expression will force to read all files, looking for matches.
      Since SonarQube 6.3, the notion of 'index files' has changed. Based on the source inclusions/exclusions, all files will be indexed, even the ones that don't match any of the languages handled explicitly handled by the sensors. As a consequence, binary files can be indexed.
      The current issue filters described above will try to read the files using the filesystem's encoding, which will typically fail for binary files.

      Possible solutions

      • Ignore errors reading the files while processing the issue filters and just show debug log, meaning that in case of an error reading a binary file, no exclusions based on the contents are applied to it.
      • Lazily compute exclusions when an issue is first created for a file. This should fix all errors (as issues can only be created for a file if it can be read) and should also bring performance improvements, as we read the file at the same time as the sensor (should benefit from OS caching) and we skip all files without issues. The downside is that this solution might be a bit more complex (to be evaluated), and also may affect performance tests of Sensors (but they probably don't use issue exclusions)
      • Only apply exclusions in the end, to the files which contain issues. This is similar to the previous point, it could be simpler to implement but it would force us to read again all issues from disk, process them, and write them back.
      • Add a required filename extension parameter to issue filters that need to read content. It's reasonable to assume that filters based on content make sense for files with specific filename extensions only. However, this would be a functional change, and so cannot be released as a bugfix.

      Workaround

      Note that a workaround to this issue is to exclude the problematic files from the analysis.

      This problem was reported here: https://groups.google.com/forum/#!topic/sonarqube/ouzdNZ4uf9U

      2017-03-16T19:47:05.9166320Z ##[error]ERROR: Error during SonarQube Scanner execution
      2017-03-16T19:47:05.9166320Z ##[error]java.lang.IllegalStateException: Unable to read the source file : 'D:/BuildAgent1_V2/7/s/<snip>/Content/Kendo/styles/Metro/editor.png' with the charset : 'UTF-8'.
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.issue.ignore.scanner.IssueExclusionsLoader.execute(IssueExclusionsLoader.java:69)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.phases.AbstractPhaseExecutor.initIssueExclusions(AbstractPhaseExecutor.java:94)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:72)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:175)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:262)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:257)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:255)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:247)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.task.ScanTask.execute(ScanTask.java:47)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.task.TaskContainer.doAfterStart(TaskContainer.java:86)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:143)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:128)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:118)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:117)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:62)
      2017-03-16T19:47:05.9166320Z ##[error]at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      2017-03-16T19:47:05.9166320Z ##[error]at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at java.lang.reflect.Method.invoke(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
      2017-03-16T19:47:05.9166320Z ##[error]at com.sun.proxy.$Proxy0.execute(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:233)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:151)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonarsource.scanner.cli.Main.runAnalysis(Main.java:110)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonarsource.scanner.cli.Main.execute(Main.java:74)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonarsource.scanner.cli.Main.main(Main.java:61)
      2017-03-16T19:47:05.9166320Z ##[error]Caused by: java.nio.charset.MalformedInputException: Input length = 1
      2017-03-16T19:47:05.9166320Z ##[error]at java.nio.charset.CoderResult.throwException(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at sun.nio.cs.StreamDecoder.read(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at java.io.InputStreamReader.read(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at java.io.BufferedReader.fill(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at java.io.BufferedReader.readLine(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at java.io.BufferedReader.readLine(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at java.nio.file.Files.readAllLines(Unknown Source)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.issue.ignore.scanner.IssueExclusionsRegexpScanner.scan(IssueExclusionsRegexpScanner.java:83)
      2017-03-16T19:47:05.9166320Z ##[error]at org.sonar.scanner.issue.ignore.scanner.IssueExclusionsLoader.execute(IssueExclusionsLoader.java:64)
      2017-03-16T19:47:05.9166320Z ##[error]... 29 more
      2017-03-16T19:47:05.9166320Z ##[error]ERROR:
      2017-03-16T19:47:05.9166320Z ##[error]ERROR: Re-run SonarQube Scanner using the -X switch to enable full debug logging.
      2017-03-16T19:47:05.9166320Z ##[error]The SonarQube Scanner did not complete successfully
      2017-03-16T19:47:05.9166320Z 15:47:05.791  Creating a summary markdown file...
      2017-03-16T19:47:05.9166320Z ##[error]15:47:05.807  Post-processing failed. Exit code: 1
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                duarte.meneses Duarte Meneses
                Reporter:
                duarte.meneses Duarte Meneses
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: