If a problem only occurs randomly once in every N times on average, how many tests do I have to perform?
I know that you can never be 100% sure, but is there a method to determine an appropriate number of tests?
I'm going to take a different approach than statistics (though I think the other response answers your actual question more directly). Any time I've encountered "a problem that only happens some of the time" as either QA or a support role it's been an investigative exercise about narrowing down why the event happens irregularly or in what situations it occurs.
Some (but certainly not all) points of investigation may be:
- Specific accounts or data.
- Differences in hosts/environments the applications or services are running on.
- Different versions of the application or service running on different hosts
- Certain days, dates, times of day or time zones.
- Certain users and their specific means of accessing the application (physical device, browser, network connection)
This sort of situation is where reproduction steps and other details from the person reporting the problem can be so valuable in resolving their issues. Telling a customer "your problem is fixed" when you're just making an educated guess can spiral in a negative direction if your experiment is based on incorrect assumptions. In my experience it's better to try to coach them about what information will help resolve their problem and how they can help you get it.