Is there any tool to determine if a test is flaky?
I'm thinking about a new approach to include autotests to CI/CD pipeline and my thought is to determine if a test is flaky before they are in pipeline.
In this case, the only way to do that is to
- setup expected test success rate;
- launch every test N times (I suppose that would be 100-300) and gather success rate;
- if test's success rate >= expected success rate, then the test can run in CI, if not - the test should be fixed or even redesigned.
I think if the test has a success rate of 98% (everyone can choose his own rate-limit) and that test can bring a real profit when launched in CI/CD.
Right now I'm working on the ways to bring that scheme to life and I'd like to check the success rate of the test every time new test added or old test changed, so it should be fast (I know some options how to do that, but they still not fast enough and by fast I mean maximum test's run time). The first thing that comes to mind is to launch the whole set in parallel mode N times and then gather rates.
But, before I'll start to do that, I'd like to know:
Are there any tools that allow you to check the test success rate (or call it flakiness) before the test is on the CI pipeline?
briley last edited by
In similar circumstances I simply used another CI pipeline as the tool, the main reasoning being that re-using of the same execution environment should give the closest results to reality.
You can set up another pipeline with the same settings but running on a single build or even a pre built version so you can focus on the tests part only.