What kind of release test report do you deliver for Agile software releases



  • Our organisation and clients expects us to give a sort of test "certificate" for each release. Each release consists of a couple of three/four weeks sprints, including testing efforts as: Automated unit-tests Automated integration-tests Exploratory testing sessions (during the sprint) Manual end-to-end test (each sprint) Manual regression suites (each release) What kind of release test report do you deliver to an enterprise client keeping in mind Agile principles like: Working software over comprehensive documentation I have looked at "traditional" testing reports and they include documenting any findings during testing. Also I have seen test reports including screen-shots to prove the testing happened. This all feels like a great waste of time, because there is a very big chance no-one will read them ever. So my question is "What do we (minimally) report to give the client the feeling we do thoroughly testing and satisfy critical managers who expect test reports?"



  • This all feels like a great waste of time, because there is a very big chance no-one will read them ever. I hear you on this one. I have wasted far too many hours of my life producing documents that nobody ever read - enough so that at one point, I devised Anna's Chocolate Bar Test. (Somewhere in the last half of the document, you state that you have chocolate in your desk for the determined few who have read this far. Then you see who comes to you for treats. Obviously this is not one to use on client documentation!). However. The agile manifesto also states "while there is value in the items on the right, we value the items on the left more". That does not mean that documentation is something you should avoid, only that you should not allow creation of documentation to take the place of creating working software. Sometimes documentation is actually a crucial part of the deliverables for your product (in a regulated industry, for example), and that's fine. There is absolutely no contradiction between a good Agile process and creating necessary documentation. In this case - I can see your difficulty because you are not (for some reason) being allowed to discuss the client's needs with the client, so it will not be tailored to their needs. But you still have to produce something that they will like and that conveys all the work you have done. I've a couple of suggestions for reading: 1) Read up about low tech testing dashboards - this is something I have not found the need to use much myself (yet), but I think it's a good match for an Agile process, especially on an ongoing project which lasts through a number of sprints. Here's how Marlena Compton implemented this in an Agile environment at Atlassian. 2) I think it's also worth reading Michael Bolton's posts about telling the testing story. For myself, the reporting I do on a release basis is aimed at a slightly different goal: I want teams to have conversations, to understand if anything they are releasing might combine in a bad way with something another team is doing. I want our release managers, should there be an issue, to be able to make decisions, and know who to go to for more information. What we produce is a two page document - a table summarising the main areas affected by this release, which team worked on it, what percentage of our customers use that functionality, and some comments on what we think the main risks are, and what we've done to investigate them. This is useful both for the teams themselves, and the release managers. (I'm also sure it will change plenty over time - this is a snapshot!) I don't think my current approach would necessarily work well in other environments, as it depends very much on a culture of transparency. Sometimes companies have a culture where they pretend that risks they don't talk about don't exist, so you shouldn't talk about a risk unless you have mitigated it. That's I think why traditional test reporting often seeks to obfuscate risks in an ink-cloud of dodgy metrics. Only you can tell whether either your company, or your client, might have that culture.



Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2