What are basic metrics of coverage with functional UI automation testing



  • There is web application product which is tested by test automation (selenium webdriver based test project in scope of the whole project) Client is asking what is the test coverage by automation of our product?

    The question is what is the best way to track, to make some docs, etc for coverage metrics, that can be used not only by client, but also by developers if they want, for example - to launch only some corresponding bunch of test scenarios based on that metric coverage ?



  • Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!"

    When you're looking at unit tests, then it's possible to use code coverage as an indicator (there are tools that can measure what percentage of the lines in your code are exercised when the test suite is run. However, you should check out this StackOverflow answer on code coverage as a metric (it's not as simple as it sounds): https://stackoverflow.com/a/90021/471206

    You're not looking at unit tests. You're looking at functional automation, at UI level. You could aim to cover every line of code via the UI, but that would be pointless, wasteful and slooooooooooooooooowwwwwww. At that level, you should probably be thinking about ... well, functionality. And there are quite a lot of different ways you could choose to model that. How will depend on what your client is really asking for.

    There's an excellent pair of articles that explain why coverage isn't so simple: Got You Covered and Cover or Discover.

    What I would suggest, is that you report several things to your client:

    Your coverage against unit and integration tests - though I would explain that 100% coverage isn't desirable, so that they understand you're aiming to test the important stuff.

    And your coverage against functional level tests, against the model you've chosen to use. That could be that you've identified the main workflows in the app, and have chosen to use your UI test suite to cover those. It could be some other model. Without a bit more information, it's hard to suggest what might be a good one (and that's also dependent on what's important to your client, so you do need to have a conversation with them too).

    Either way, you'll need to explain why you chose that model, and why you think the results you achieved are a good balance of cost vs value.



Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2