A language plugin should be able to log issues on any file

Context

Currently, a plugin can request the indexing of files with certain extensions, e.g. .java, and the platform will index those files and hand them off to the requester. However only a single plugin may register a given extension, and files with extensions that have not been requested by any plugin are simply unavailable (unless the indexing of all files has been enabled). As a consequence, plugins cannot create issues on random files.

Purpose of the MMF

A language plugin should be able to analyse any file that exists in the project - whatever the extension of that file, and create issues on those files.

Concrete example:

• The Java plugin wants to be able to create issues on the pom.xml or web.xml files if they exist in the project
  • The Java plugin should be able to work on *.xml files without causing a conflict with the XML plugin
• The PHP plugin wants to be able to create issues on the config.ini file if it exists

Proposal

• Index by default all files that are part of source sets, not just the ones matching file suffixes declared by Language extension points (same as using sonar.import_unknown_files=true)
  • continue to honor sonar.sources/sonar.tests/sonar.exclusions/... -> update scanners if needed to have more files in the source sets (like src/main/* in Maven)
• To avoid reading content of all indexed files (performance penalty + possible failure on binary files):
  • delay initialization of file metadata (lines / charset / offset per line). The first sensor trying to access one of those attribute will trigger initialization. Have a "secret" property to revert to systematic metadata initialization to not break language team performance ITs.
  • only blame / compute CPD on files to be published (see next item)
  • optimize some FS query (and introduce dedicated predicates). It should be cheap to query by file extension or by filename.
• To not bloat scanner report with too many files, only publish:
  • "touched" files = files for which a Sensor had stored some data (measure/issue/highlighting/...)
  • files could be explicitly marked for inclusion in the report using a new API (e.g. sensorContext.markForPublishing(inputFile)): this is to avoid a file without language to disappear when all issues have been fixed

Example for the JAVA and PHP plugins

• On 6.3 with non-upgraded plugin: FS query will return an InputFile (assuming it is part of scanner source sets). If at least one issue is saved, the file will be published.
• On 6.3 with upgraded plugin: Java plugin can force publishing of all **/pom.xml using new API to have better consistency (number of files in the project will not change depending on if an issue is raised or not)

for (InputFile f : fs.inputFiles(p.matchesPathPattern("**/*.xml")) {
  if (f.path().endsWith("pom.xml") {
    if (context.getSonarQubeVersion().isGreaterThanOrEquals("6.3")) {
      context.markForPublishing(f);
    }
    executePomChecks(f);
  } else if (f.path().endsWith("ejb-jar.xml")) {
    if (context.getSonarQubeVersion().isGreaterThanOrEquals("6.3")) {
      context.markForPublishing(f);
    }
    executeEJBChecks(f);
  }
  ...
}