Tool to minimize tests number and maximize code coverage
briley last edited by user
I am currently undergoing black-box testing campaigns for applications that require file input in linux environments. For that I need sets of test files of different formats that can be used to be passed as input to the different applications I am interested in.
To monitor the code coverage for a specific set of testing files, I am using lcov/gcov. The problem is that a set of files that has decent code coverage can become rather large, therefore taking a lot(and I mean a lot) of time to complete the tests using such a set. So I would like to be able to minimize such a set, while keeping the same code coverage (in other words eliminate the redundant test files that stress the same part of an application).
I tried using the minset tool from the peach fuzzing framework, which would be exactly what I would need but unfortunately it can be only used on Windows and as far as i have tried cannot be ported on linux-like operating systems.
So, do you guys know any tool that could do what I have described above. Or perhaps have an idea on how I could achieve that. Thanks a lot.
You could implement your own version of minset that would be useful to anyone that uses gcov/lcov or, with a small modification, also to users of other code coverage measuring tools.
Export gcov/lcov report to some processable format, e.g. XML. The report should describe coverage per method/class/package (granularity is up to you) for each test you executed and its execution time, e.g., as a quadruple:
(test, package, coverage, executiontime).
Describe it as an optimization problem and find corresponding algorithm that will work on your exported data. The optimization problem is that by selecting a subset of tests you want to
- maximize the test coverage per package (or method, class),
- while minimize/limit total test execution time to some value.
This sounds a bit like a knapsack optimization problem, where you try to limit the total mass (here test execution time) of items put into a knapsack while maximizing their total value (here test coverage). The only additional requirement is you are trying to maximize local value (coverage per package). But I'm sure there are such variations of the knapsack problem and so proper algorithms.
Stackoverflow users may be of more help here or authors of minset method in the peach fuzzer tool you mentioned.