Uploaded image for project: 'SonarCFamily'
  1. SonarCFamily
  2. CPP-3128

Test rules with all supported C++ standards



    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.25
    • Component/s: C++
    • Labels:



      The quality of the CFamilyPlugin should be consistent across C+
      + standard language version and releases:

      • The use of new a standard does not invalidate older rules, and they should preserve their behavior
      • C++ users are not necessary using the newest standard and we should make sure that the product works for them, according to CppDevSurvey-2021:
        • 59.77% are not allowed to use c++20
        • 21.06% are not allowed to use c++17
        • 8.09% are not allowed to use c++14
      • Clang updates (and modernization) may impact the behavior of AST matcher, and break existing rules:
        • Matcher behavior is changed (e.g. traverse)
        • The generated AST Tree is different between language versions (e.g. MaterializeTemporaryExpr)
        • Compiler bugs are fixed, changing behavior
        • DR changes are retroactively applied to older standards
      • Currently, our lit test is checked against only the newest version if no behavior change is intended:
        • No automated test for lack of semantic change
        • Rules implementation get changed to use new matches, fix bug


      • Improve the testing infrastructure by enforcing consistency over different standards.
      • Run lit test in each supported C++ standard version:
        • Supported means that messages should be produced for violations
      • Integrate into Pull Request hooks of the CI
      • Preferably one invocation to run with all possible versions
        • This should be the default mode for lit invocation
      • Opt-in quick/simple mode (to reduce run time on development):
        • Only C++20 (newest standard)
        • Newest supported by file (current)
        • Not enabled by default
      • Supporting multiple C/Obj-C standards is out of scope


      The goal of “Run lit test in each supported C++ standard version” requires an answer to the following questions:

      1. How to define supported standards for a specific test.
      2. How to extend testing infrastructure (lit) in a way, so each test is invoked multiple times with each supported standard.
      3. How to handle checks messages, that behave differently between standards.
      4. How to approach testing new features with older rules (e.g. structured binding with C++11 rule)

      1. Define supported standards for each test:

      • Define supported standard using available features, e.g.:
        • Required: C++11 - test need C++11 features, so will work with C++11 or newer standard
        • Unsupported: C++17 - test should not be run when C++17 features are enabled (e.g. superseded by new rule), this would also disable newer standards like C++20, that also provide C++17 features
      • Manually update each test to define available features:
        • All standards supported by default (no requirements)
        • Without updates, parsing errors for the too old standard would be produced, leading to false failures
      • Incrementally introduce changes:
        • Implement required changes to test running infrastructure (lit, Tester)
        • Update all tests in batch, to run only in the same standard as used currently:
          • For a test with --std=version, configure that version
          • For all other test defaults for C++20
        • Incrementally, refactor and update test in batches, to avoid large pull-request

      2. Update lit tester to run each test in all C++ standards:

      • Use test-format to duplicate runs, for each standard mode:
      • Generate available features listing standard and previous, e.g. for C++17: C++98, C++11, C++14, C++17
      • Make standard version configurable by using substitution:
        • %stdver expanding to --std=c++11 or --std=c++17
        • Extend %checker substitution to include the version
      • Command-line option to override the list of used standard modesAdd substitution for %stdver

      3. Messages:

      • Extend CHECK syntax to support specification of standard for which message should appear:
        • CHECK - always appear (no filter)
        • CHECK[cpp17] - only C++17
        • CHECK[cpp17+] - from C++17
        • CHECK[cpp11, cpp14] - for C++11 and C++14
        • CHECK[cpp11-17] - for C++11, C++14 ,and C++17
      • Allows single test for different messages in different standard
      • Adjust expected messages to print in which standard mode it fails

      4. Testing old rules with C++ features introduced in new standards:

      • Use preprocessing directives and __cplusplus macro to guard the use of new features
      • Refactor the test to group code requiring a newer standard together to reduce the impact of readability


          Issue Links



              tomasz.kaminski Tomasz Kamiński
              tomasz.kaminski Tomasz Kamiński
              0 Vote for this issue
              2 Start watching this issue