How long should test results be kept?
We've got Jenkins jobs running Unit Tests, Integration Tests, JMeter Tests, and Selenium Tests. Given my choice I'ld keep all these test results forever, but maybe that is excessive.
- How long does your team keep test results?
- How far back in history have you found useful?
- Have you run into any performance issues related to keeping too much history?
- Do you archive test results outside of Jenkins? How?
As always, it depends. If there are regulatory requirements, results need to be kept for as long as the regulations state. Otherwise, it's a business decision.
1. How long does your team keep test results? In my experience this depends on a number of factors. I've been in places where test results for new development would be kept indefinitely, but test results for regression were regarded as ephemeral. I've also been in places where the full logs were discarded or archived, but a summary was retained indefinitely.
2. How far back in history have you found useful? Again, it depends. Mostly I've found that if something goes back more than 3 major versions, looking for more resolution is a waste of time. Occasionally it's been necessary to go back to the test case documents from the original implementation to determine whether something was covered initially. If regressions are reported as soon as they're observed, the reporting should include the first known failing build and the last known good build. Ideally these are sequential, but this isn't necessarily the case.
3. Have you run into any performance issues related to keeping too much history? This is one of the few that doesn't get "it depends" - disk space is limited, and logs accumulate. The process I prefer to use is to automatically archive logs off to a pre-defined location at least once a day, including application logs where appropriate (at one employer this was essential if you didn't want to be dealing with trying to open 4GB text files - I created the daily archive task because I was sick of trying to access immense log files). The archive location stores the logs without compression for a period of time, then zips logs older than the agreed time-frame. We still needed to clean out the oldest logs periodically and ultimately decided that if it was older than a year, we didn't need to keep it. We versioned our test script code, so it was possible to rerun old code against old versions of the application if necessary.
4. Do you archive test results outside of Jenkins? How? I've used a task that would export the logs to a datestamped directory on a daily basis. It's also possible to do it per build with a datetime stamp on the directory. Zipping via command line is easy to do, so that was also part of the task. The naming convention was usually %testsuitename%%YYYY%%MM%%DD%[%hh%%mm%%ss%] for the directory or zip file.