Testing when two bugs cancel each other out?
I'm writing test cases for a simple open-source utility using Perl's Test::More module. There are two bugs in the utility that, due to the way they interact, cause one of the test cases to pass. (I discovered the two bugs because there are other test cases where the two bugs don't cancel each other out.) If either bug is fixed (but not both), suddenly that test case starts failing. How should I annotate the test to deal with this situation? Normally when there's a bug I simply mark the test as an expected failure with a reference to the bug report. In this case, I can't mark it as an expected failure because the test itself is not failing. My default plan is to leave the test marked as expected to pass and just add a comment next to the test case explaining the situation. There are a few reasons why I don't like this: The passing test case gives a false sense of proper behavior. When a developer fixes one of the two bugs they might be surprised to see a newly failing test case. The comment next to the test case should make it easy to investigate the situation and change the test from expected pass to expected fail, but I still prefer to avoid surprises. When a commit intended to fix a bug causes a test to start failing, others will be suspicious of the bug fix (and the associated change to mark the test case as expected to fail). Suppose commit #1 fixes one of the bugs and marks the test case as expected to fail. Commit #2 fixes the other bug and marks the test case as expected to pass. Now commit #2 can't be cherry-picked to a release branch without either cherry-picking commit #1 or re-marking the test case as expected to fail. Cherry-picking commit #1 might be considered too risky for the release branch, but re-marking the test case increases the likelihood of introducing a merge conflict resolution error. The Test::More module (and the Test Anything Protocol) don't support marking a test as "accidental pass", but even if they did I don't know if that would be the right way to deal with this case. How should I mark the test case to reduce the risk of future problems?
Ok, I edited this answer based on your clarified question. (It makes sense now.) It is important to treat automation as a tool rather than a "tester". A major fallacy in automation is to treat each of the test cases as pass/fail tests. Instead, it is better to treat them as "trip wires" that simply indicate additional attention should be paid to the test case that is failing in the automation. The tendency to treat them as pass/fail tests is bound in the (improper) use of the automation results as a metric. As you have shown in your question, any metrics derived from the automation results can be very misleading. As an extension of that, I would suggest not generating defects based on automated results. Instead the defects should be generated only after the "failure" has been shown to be significant through manual verification. My suggestion is to not worry about the canceling bugs, since they still show up individually. Hope that is a better answer!