Should automated checks be agnostic of software configuration files?
Let's say we have a piece of Software, with a bunch of constants in a configuration file that sets certain arbitrary values, like a timeout of 300 seconds.
Now we have a script for automated checks in form of unit tests of the software, one of them being a check for if the timeout works properly.
Most of the time, I see the timeout value being hardcoded as a magic number into the test script, as the developers try to separate software from tests as much as possible.
But in my opinion, the configuration of the software is not part of the software itself, as the values are arbitrary, and thus not only could be used, but actually should be used by the tests.
The advantage would be, that if the timeout values change, no test would need to be adapted at all.
But all developers are reluctant to do that, any idea why?
Alberto last edited by
When the same value is checked all the time autotest becomes nearly useless (no new knowledge of the product). Unless you check that the default value of timeout is set correctly in config-file. And in this case configuration value could not be used by tests and your devs are right.
But you are checking that the timeout works properly so it's not good for your autotest to check only the value provided by default.
If you have one check of this timeout in your test suite it would be preferable to check a randomly set timeout within a range of valid values. And after the test reset timeout to the system default value (300 seconds) or to the value before test (cause generally autotest should keep software in the same state as it was before test, if they are not integrated in dependent sequence of tests).
If you have two check, another one could be done with invalid value (if the system supposed to handle this case somehow).
There could be even more checks e.g. if error in this configuration feature has high impact on company reputation.