Assertion coverage and assertions count in industry



  • I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

    I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

    In research papers I found two related metrics:

    • assertion coverage/checked coverage: percentage of statements directly covered by the assertions, proposed in https://www.st.cs.uni-saarland.de/publications/files/schuler-icst-2011.pdf
    • assertions count proposed in http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1079.4905&rep=rep1&type=pdf .

    but that's research and I couldn't find any tooling for measuring/reporting them for Python.

    Are those metrics used in industry?



  • One alternative is mutation testing.

    From the https://pitest.org/ :

    Mutation testing is conceptually quite simple.

    Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived.

    The quality of your tests can be gauged from the percentage of mutations killed.

    Pitest is for JVM, but most main programming languages have mature tools for it.

    For more details, I suggest https://www.youtube.com/watch?v=LoFJajoJQ2g .




Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2