Regression testing for Network Equipments like Router/Switches
I joined a new company that is working on network equipments like Routers/switches.
We have a product(a network switch) in end stage of development. This supports MPLS, OSPF, IS-IS and BGP protocols. We have an existing regression testcase suite to run on daily builds.
My question is related to analyzing the test results. We have seen that some of the testcases have different results (Pass/Fail) when run on the same build and with same inputs.
What do you make out of it?
- The product is very buggy.
- The results do vary for testcases that involve network products. So don't bother??
Ah, the bane of every tester.
I should first remind you that you should control every factor to make them repeatable - which is a lot of effort. There's the environment (OS's, machines, connections/interfaces between them, then the application under test). Test fixtures are vitally important - setting the whole environment to a fixture in this case.
But you may find that instead of controlling all those factors, if you can - run the same set of tests a number of times and take a probabilistic view to comment on the results.