How to automatically rerun failed TestNG tests on Jenkins and update the final results to pass if retries passes?

Question

How to automatically rerun failed TestNG tests on Jenkins and update the final results to pass if retries passes?

Description: I have two tests that ran. Test1 passed, but Test2 failed then on retry it passed.

Actual Results: Tests run: 1, Failures: 1, Errors: 0, Skipped: 0.

Expected Results; Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

or

Tests run: 2, Failures: 0, Errors: 0, Skipped: 1 (Skipped for retry tests)

    public class TestRetry implements IRetryAnalyzer {
    private static int MAX_RETRY_COUNT = 1; 
    //Only works for value of 1; Anything more than that causes 
    infinite retries for the same test even when it passes.

    AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);

    public boolean isRetryAvailable() {
        return (count.intValue() > 0);
    }

    public boolean retry(ITestResult result) {
        boolean retry = false;
        if (isRetryAvailable()) {
            Reporter.log("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT, true);
            retry = true;
            count.decrementAndGet();
        }
        return retry;
    }
}

  public class TestListenerSupport extends TestListenerAdapter {
  TestUtilities testUtil = new TestUtilities();

@Override
public void onTestStart(ITestResult result){
    //Handle events before test start
    //Obtain specific annotation info from test methods
    Stable stable = 
  result.getMethod().getConstructorOrMethod().getMethod()
  .getAnnotation(Stable.class);
   Maturing maturing = 
  result.getMethod().getConstructorOrMethod().getMethod()
  .getAnnotation(Maturing.class);
   Preflight preFlight = 
   result.getMethod().getConstructorOrMethod().getMethod()
   .getAnnotation(Preflight.class);
      if (stable != null){
           Reporter.log("Annotation is set to STABLE! for '" + 
    result.getMethod().getMethodName() + "'.", true);
      }
      if (maturing != null){
           Reporter.log("Annotation is set to MATURING! for '" + 
    result.getMethod().getMethodName() + "'.", true);
     }
      if (preFlight != null){
           Reporter.log("Annotation is set to PREFLIGHT! for '" + 
    result.getMethod().getMethodName() + "'.", true);
    }
}


  @Override
  public void onTestFailure(ITestResult result){

      if (result.getMethod().getRetryAnalyzer() != null) {
            TestRetry retryAnalyzer = 
     (TestRetry)result.getMethod().getRetryAnalyzer();

            if(retryAnalyzer.isRetryAvailable()) {
                result.setStatus(ITestResult.SKIP);
            } else {
                result.setStatus(ITestResult.FAILURE);
            }

            Reporter.setCurrentTestResult(result);
        }
}

@Override
public void onTestSkipped(ITestResult result){

    Reporter.log("onTestSkipped invoked! " + result.getThrowable().toString(),true);
    Reporter.log("onTestSkipped Exception Message: " + result.getThrowable().getMessage(),true);
    Reporter.log("onTestSkipped Exception Cause: " + result.getThrowable().getCause(),true);
    Reporter.log("onTestSkipped Exception StackTrace: " + result.getThrowable().getStackTrace(), true);
    AppiumDriver driver = getDriverInstance(result);
    if(driver!=null){
        TestUtilities testUtil = new TestUtilities();
        try {
            testUtil.getScreenShot(driver, result.getMethod().getMethodName() + "_Skipped");
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        Reporter.log("Killing appium driver instance",true);
        driver.quit();
    }
    else
        Reporter.log("onTestSkipped - null driver detected", true);
}

@Override
public void onTestSuccess(ITestResult result){
    Map map = new HashMap();
    //get the testcase IDs from input xml file used for creating a test run in TestRail
    Map<String,String> test_case_id = result.getMethod().getXmlTest().getAllParameters();
    String test_name = result.getMethod().getMethodName();
    testUtil.writeResult("1", test_case_id.get(test_name));

}

Answer 1

TestNG provides the IRetryAnalyzer interface, which you can implement like this:

public class MyRetryAnalyzer implements IRetryAnalyzer {

    private AtomicInteger retries = new AtomicInteger(3);

    @Override
    public boolean retry(ITestResult result) {
        return retries.getAndDecrement() > 0;
    }

}

Afterwards, you can use it as follows:

@Test(retryAnalyzer = MyRetryAnalyzer.class)
public void myTest(){
    // ...
}

However, flaky tests are evil—and so is the excessive use of retry rules. Have a look at this related question:

How to deal with flaky tests that have intermittent failures?

Answer 2

I adjusted my code with what Nick mentioned in here https://stackoverflow.com/questions/11622365/testng-how-to-only-output-one-pass-fail-result-when-test-is-rerun

and it worked.

Thanks.

public class RetryAnalyzer implements IRetryAnalyzer {
private int count = 0;
private int maxCount = 2;

@Override
public boolean retry(ITestResult result) {
if (!result.isSuccess()) {
  if (count < maxCount) {
    count++;
    result.setStatus(ITestResult.SUCCESS);
    String message = Thread.currentThread().getName() +
    ": Error in " + result.getName() + " Retrying "
        + (maxCount + 1 - count) + " more times";
    System.out.println(message);
    Reporter.log(message);
    return true;
  } else {
    result.setStatus(ITestResult.FAILURE);
  }
}
return false;
}
}