Stability testing for open source operating systems
You can find quite an interesting panoply of open source operating systems to choose from. Being open source you can have access to a lot of their code, build systems, bug tracking, etc.
As of this time I didn't came across a document that plans or shows the design of stability testing for an OS. It would be useful for me to know about: what test suite they use, what benchmarks, what stress tests, do they have some stability test plans released to the public, etc?
I used here stability testing as an umbrella term for:
- reliability testing / endurance Testing (MTBF, MTTF)
- stress testing
- load testing
- recovery testing
- low resources simulation
- or any other testing that would qualify in "Performance Testing" category.
Can someone provide some guidelines on how open source OS stability testing is done?
emmalee last edited by user
First thing to realize is what open-source means for software: the basic idea is that the source code is available. In more specific terms, someone has decided (for a new project or a running one) to share the code, i.e. every technical detail with the whole world. Furthermore, since there's technically nothing that can stop people from compiling and/or fully using such code it typically also means that the software is free (as in free beer) to use.
That's about the idea of open source. It has nothing to do with planning or organization of human resources.
So in practice, and if the project is successful and interesting enough, the above will also mean that there will be people who will actually (try to) use it, and there will be people that will look at that code, study it, analyse it, play with it and talk about it, and eventually send bugfixes back to the author.
From this list of activities, at least three involve, or are a form of what is known as software testing.
So vaguely said, the quality in FOSS comes from:
- lot of discussion before design and coding
- perpetual code peer review
- feedback from people trying to play with the the SW
- feedback from people trying to use the SW in real situations
- feedback from people trying to help develop the SW (maybe just fix their use case)
- sometimes even feedback from professionals paid by companies that have their business built on FOSS technologies
- no deadline tyranny
- and also quality of the original code base and architecture
much more than creation and execution of testing plans (perfectly ballanced to match the needs of the focus group).
Other thing to consider is, that each planning must start with a strategy. For commercial products, strategy is money and money is strategy: Who will run it? How will they run it? How much will get back at us? How much support requests we'll be able to handle? How many different hardware platforms even exist?
The strategy part is really a tough call since it usually happens before anything else, and the decisions taken here will influence almost every aspect of the whole project, and are extremely expensive to change. Therefore great amount of information needs to be gathered, which costs additional resources.
No need to say that any reasonable test planning needs to copy the strategy. (You're not going to test the brand new .NET-based accounting software with Powershell-powered installation routines on Win 2000, are you?)
Since most open source starts either as "scratching someone's own itch", without much focus on how much other people have it, it's virtually impossible to take this step, but it's also much less important, since more users can't scratch itch any better than the developer can themselves. If the scratcher is not sharp enough, they can help sharpening (send a patch).
Just in case you got that feeling: I'm not trying to say that there cannot be a highly organized group of professionals proceeding with a rigorous and well-thought testing plan on an open source. It definitely can and many such teams definitely do exist (I think Red Hat might have group like that for Fedora). But just because they are open-sourcing the code, it does not mean that they are willing to open-source their know-how.
At the end of the day, what counts most are bug reports, test cases and patches.
Operating systems are no exception here.