How can I find the weakness of our "Quality Assurance Process"?
We have a very huge product. Although we had all kinds of tests (units test, integration tests and automation tests), there are still lots of bugs and the customer complains a lot about it.
My manager has asked me to figure out why we are failing to provide a good product.
Please notice, here I am talking about product defects. Thus, even though process and project are the other aspects of quality, I am focused here only on the product.
As a potential solution, I suggested to collect a random sample of 50 tickets (bugs) and study them carefully, so that we figure out why we did not detect them during the development phase.
But, what else shall we do to debug our QA process?
I would start by collecting cleaning up the data - don't pick 50 random bugs, but start by classifying them, manually or semi-automatically, maybe using keywords or information from the bug's logs - bug's age, environment, software module, etc. A bug can end up belonging to more than one category.
Move on and look for correlations in the data. Again this might be a semi-automated process using ad hoc scripts or Excel sheets.
And now comes the fun part. Pick a few representative bugs from each category or correlation, and dig down to find the underlying problems.
"Problems" and not "a problem", because it's best to ask why more than once. Five times is usually a good choice.