Uploaded image for project: 'SonarCFamily'
  1. SonarCFamily
  2. CPP-3128

Test rules with all supported C++ standards

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.25
    • Component/s: C++
    • Labels:
      None

      Description

      Why

      The quality of the CFamilyPlugin should be consistent across C+
      + standard language version and releases:

      • The use of new a standard does not invalidate older rules, and they should preserve their behavior
      • C++ users are not necessary using the newest standard and we should make sure that the product works for them, according to CppDevSurvey-2021:
        • 59.77% are not allowed to use c++20
        • 21.06% are not allowed to use c++17
        • 8.09% are not allowed to use c++14
      • Clang updates (and modernization) may impact the behavior of AST matcher, and break existing rules:
        • Matcher behavior is changed (e.g. traverse)
        • The generated AST Tree is different between language versions (e.g. MaterializeTemporaryExpr)
        • Compiler bugs are fixed, changing behavior
        • DR changes are retroactively applied to older standards
      • Currently, our lit test is checked against only the newest version if no behavior change is intended:
        • No automated test for lack of semantic change
        • Rules implementation get changed to use new matches, fix bug

      What

      • Improve the testing infrastructure by enforcing consistency over different standards.
      • Run lit test in each supported C++ standard version:
        • Supported means that messages should be produced for violations
      • Integrate into Pull Request hooks of the CI
      • Preferably one invocation to run with all possible versions
        • This should be the default mode for lit invocation
      • Opt-in quick/simple mode (to reduce run time on development):
        • Only C++20 (newest standard)
        • Newest supported by file (current)
        • Not enabled by default
      • Supporting multiple C/Obj-C standards is out of scope

      How

      The goal of “Run lit test in each supported C++ standard version” requires an answer to the following questions:

      1. How to define supported standards for a specific test.
      2. How to extend testing infrastructure (lit) in a way, so each test is invoked multiple times with each supported standard.
      3. How to handle checks messages, that behave differently between standards.
      4. How to approach testing new features with older rules (e.g. structured binding with C++11 rule)

      1. Define supported standards for each test:

      • Define supported standard using available features, e.g.:
        • Required: C++11 - test need C++11 features, so will work with C++11 or newer standard
        • Unsupported: C++17 - test should not be run when C++17 features are enabled (e.g. superseded by new rule), this would also disable newer standards like C++20, that also provide C++17 features
      • Manually update each test to define available features:
        • All standards supported by default (no requirements)
        • Without updates, parsing errors for the too old standard would be produced, leading to false failures
      • Incrementally introduce changes:
        • Implement required changes to test running infrastructure (lit, Tester)
        • Update all tests in batch, to run only in the same standard as used currently:
          • For a test with --std=version, configure that version
          • For all other test defaults for C++20
        • Incrementally, refactor and update test in batches, to avoid large pull-request

      2. Update lit tester to run each test in all C++ standards:

      • Use test-format to duplicate runs, for each standard mode:
      • Generate available features listing standard and previous, e.g. for C++17: C++98, C++11, C++14, C++17
      • Make standard version configurable by using substitution:
        • %stdver expanding to --std=c++11 or --std=c++17
        • Extend %checker substitution to include the version
      • Command-line option to override the list of used standard modesAdd substitution for %stdver

      3. Messages:

      • Extend CHECK syntax to support specification of standard for which message should appear:
        • CHECK - always appear (no filter)
        • CHECK[cpp17] - only C++17
        • CHECK[cpp17+] - from C++17
        • CHECK[cpp11, cpp14] - for C++11 and C++14
        • CHECK[cpp11-17] - for C++11, C++14 ,and C++17
      • Allows single test for different messages in different standard
      • Adjust expected messages to print in which standard mode it fails

      4. Testing old rules with C++ features introduced in new standards:

      • Use preprocessing directives and __cplusplus macro to guard the use of new features
      • Refactor the test to group code requiring a newer standard together to reduce the impact of readability

        Attachments

          Issue Links

          breaks down into

          Task - A task that needs to be done. CPP-3176 Update test infrastructure to support running lit test with multiple C++ standards

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. CPP-3177 Review supported standard for lit test

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. CPP-3207 LineLength: ignore the reports of LineLength rule in tester in the blocks disabled by preprocessor

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. CPP-3210 S5271: The issues is not raised since CPP17 in the presence of removed construct "throw(int)"

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. CPP-3216 S3584: inconsistient issue location highlighting lambda body for C++11&14 and lambda name for C++17&20

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. CPP-3235 S1481: False negative for C++03,C++11,C++14 modes in the presence of invalid language constructs

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. CPP-3244 Assertion failure for arrays with invalid extent (no crash/potential UB)

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. CPP-3193 Assertion failure on newer-standards language constructs (look benign)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. CPP-3209 S3719: Issue not reported for C++03 mode

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. CPP-3218 S854: Issue raised only for windows in C++03 mode

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. CPP-3212 S818(LiteralSuffix): redundant duplicate reporting on template instantiations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. CPP-3201 S3230: The rule should raise in C++03 for uncessary in-class value init and assigment in constructor

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. CPP-3208 S1774,S3358: Add exception for constexpr functions in C++11 mode

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. CPP-3180 S3471,S1016: Duplicate reports for each instantiation of a template

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. CPP-3220 S3542: Issue duplicated for each macro expansion

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. CPP-3217 S5018: unclear reporting, especially for code involving lambdas in different C++ versions

          • Major - Major loss of function.
          • To Do

          False Negative - True issue, expected to be covered by a rule, but ultimately not reported CPP-3214 S1854: False positives for deadstore detection on constructors in C++17

          • Major - Major loss of function.
          • In Progress

          False Negative - True issue, expected to be covered by a rule, but ultimately not reported CPP-3179 S5995: Issue is not raised for dependent type variable captured by copy for C++11

          • Major - Major loss of function.
          • Closed

          False Negative - True issue, expected to be covered by a rule, but ultimately not reported CPP-3194 S3400: missing trivial lambdas for C++17 and C++20

          • Major - Major loss of function.
          • Closed

          False Negative - True issue, expected to be covered by a rule, but ultimately not reported CPP-3195 S3807: False negative when passing nullptr in variadic arguments to strfmon and swprintf

          • Major - Major loss of function.
          • Closed

          False-Positive - Unexpected issue reported by a rule CPP-3196 S1236: False positive for assignment operators returning self converting to a non-Self type up to C++14

          • Major - Major loss of function.
          • Closed

          False Negative - True issue, expected to be covered by a rule, but ultimately not reported CPP-3213 S5025: Fix false negative when pointer conversion is required or new is placed in parens

          • Major - Major loss of function.
          • Closed

          False Negative - True issue, expected to be covered by a rule, but ultimately not reported CPP-3237 S864: False negative in a ternary operator context with user-defined operators in C++ before C++17

          • Major - Major loss of function.
          • Closed

          False-Positive - Unexpected issue reported by a rule CPP-3239 S2738,S1181: False positive for a try/catch-style for destructor in C++03

          • Major - Major loss of function.
          • Closed

          False-Positive - Unexpected issue reported by a rule CPP-3240 S881 IncAndDecMixedWithOtherOperators: False postive on lambda init-capture for C++11 and C++14

          • Major - Major loss of function.
          • Closed

            Activity

              People

              Assignee:
              tomasz.kaminski Tomasz Kamiński
              Reporter:
              tomasz.kaminski Tomasz Kamiński
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: