(NB: I don't have a very strong background in statistics, so I would not be surprised if this question is naive or the answer is apparent from some rather basic principle.)
I'm wondering how I might analyze the following situation using principles of statistics:
A Manufacturing Process
Consider a manufacturing process with a constant defect rate of 1%. (We get this average defect rate from many years of manufacturing experience and from customer reports of part failure in the field.) The defects are random in nature and not readily apparent. Quality control personnel have to run all manner of different tests to find them and sometimes the defect eludes all the tests used. (Maybe a certain defect is only apparent after a part is dipped in benzene, cooled to -100 C, and then hit with a rubber mallet.)
On each day, quality control personnel decide what sort of tests to use on the parts. On Wednesday, quality control testing reveals 13 defects per 1000 units by subjecting each of the parts to 25 kinds of tests. On Thursday, quality control testing reveals 1 defect per 1000 units by subjecting the parts to 25 kinds of tests (not necessarily the same tests as the day before).
Questions About Defects
Based on the known average defect rate of 1%, can we draw any conclusion about the testing results on Thursday? Can we quantify the probability that there just happen to be very few defects on this day? How would this compare to the probability that quality control has missed more defects than usual? Perhaps we need information about how effective each sort of test is, on average. Or do we always need to apply the same 25 tests on each day to draw any useful conclusions?
A larger question is: In this situation, can we draw a useful conclusion about the overall defect rate by testing only 100 parts? Suppose we find 10 defects when testing 100 parts. What's the probability that this represents a manufacturing run with an unusually high 10% defect rate? What's the probability that this represents unusually effective testing that found all defects with a smaller number of tests? What additional information is necessary to answer these kinds of questions?
