Uploaded image for project: 'Rules Repository'
  1. Rules Repository
  2. RSPEC-5973

Tests should be stable

    XMLWordPrintable

    Details

    • Message:
      * Make this test stable and remove this "successPercentage" argument.
    • Highlighting:
      Hide

      The "successPercentage" argument

      Show
      The "successPercentage" argument
    • Default Severity:
      Major
    • Impact:
      Low
    • Likelihood:
      High
    • Default Quality Profiles:
      Sonar way
    • Covered Languages:
      Java
    • Remediation Function:
      Constant/Issue
    • Constant Cost:
      20min
    • Analysis Scope:
      Test Sources

      Description

      Unstable / flaky tests are tests which sometimes pass and sometimes fail, without any code change. Obviously, they slow down developments when engineers have to rerun failed tests. However the real problem is that you can't completely trust these tests, they might fail for many different reasons and you don't know if any of them will happen in production.

      Some tools, such as TestNG, enable developers to automatically retry flaky tests. This might be ok as a temporary solution, but it should definitely be fixed. The more flaky tests you add, the more chances there are for a bug to arrive in production.

      This rule raises an issue when the annotation org.testng.annotations.Test is given a successPercentage argument with a value lower than 100.

      Noncompliant Code Example

      import org.testng.annotations.Test;
      
      public class PercentageTest {
          @Test(successPercentage = 80, invocationCount = 10)  // Noncompliant. The test is allowed to fail 2 times.
          public void flakyTest() {
          }
      }
      

      See

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              nicolas.harraudeau Nicolas Harraudeau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated: