Can I force a Jenkins job to report a successful TestNG test even when the suite has failures?
Can I force a Jenkins job to report a successful TestNG test even when the suite has failures? I have a test suite with 100 tests and I want to report a success in Jenkins if there are 3 or fewer errors. What is the best way to do this? Here are my ideas so far: Use the Jenkins "Groovy post build plugin" to force the button green after the build step already reported yellow. Run the tests in a Jenkins pre-build step and then use the TestNG XmlResult class to load the test results in the regular build step, count the fails in the result, and then return a 0 or 1 exit code? Use the Jenkins Build-Flow plugin to ask all 100 tests to re-run if 1 or more fails. I am not sure of the best way to do this and I am wondering if anyone has a better idea. My workaround for this for now is to divide my test suites into 3 separate suites that run in a fixed order: first, a point-suite with 1 test, then a smoke-suite with a handful of tests, then the full regression as long as the smoke-suite passed.
You might want to take a look at TestNG Jenkins plugin. Although it does not currently fix your problem, you will find a group of people facing the same situation as you discussing an issue. You might chime in and help develop/test this fix of the plugin.