Disabling failing tests in CI if they discover known bugs
Laycee last edited by
Number one rule with continuous integration (CI) server is to have build always green and when a test fails fix the system under test as quickly as possible. However, we have currently end-to-end and integration tests on CI server that fails because of known bugs. We are aware of those bugs but they are minor and they have less priority to be fixed immediately. Consequently, build is no longer green and if new bug will be introduced we will have no such a good feedback on that. Our first solution is to disable such failing tests, keep in the test code a link to a bug (JIRA issues) and keep in JIRA a link back to the test. This obviously have some shortcomings such as the need to track and remember about such disabled tests. Also, disabled tests will no discover other bugs. What would be your approach for such a situation?
Do not disable the tests, which fail because of known bugs. I would suggest to group the failing tests and run them in another test suite. You might use a group annotation on the tests for that, maybe "knownFailures" or so. The group annotation is better then a "link to issue" thing. Group annotations are supported by various test tools. Configure your build matrix to allow the failure of this test suite. Also exclude the group of tests (forming the failing suite) from your main test suite. That keeps the main suite green. You can then work on both suites in your matrix. The main will switch between green and red, while the "knownFailures" stays red - BUT here the number of failing tests will reduce, if you fix some of the known bugs. ,) And you may introduce an automated check: for tests with the @group knownFailures, which are green. That helps you to remove the @group annotion, when they go from red to green.