5

I have a dataset in which there are about 500 measurements for a particular scoring metric. The data are normalized so that the mean is 0 and the standard deviation is 1, but the distribution is somewhat arbitrary. Here's a histogram:

Histogram of score distribution

I have no a priori reason to expect a particular distribution either. And these data comprise the entirety of the population.

Within this set, there are 11 data points of interest, as determined by a separate metric. These are indicated by the red lines on the histogram. It appears that the scores for these points in the first metric are not random, but are instead significantly higher than average (the mean score for these 11 points is ~1.43).

How can I test that these 11 data points are not drawn randomly from the population?

Mike B
  • 53

1 Answers1

1

Assuming you're particularly interested in whether the sample mean is too different from the population mean to be consistent with random selection from the population, and you have the complete population you might look at the distribution of sample means for samples of size 11 from that population.

This is the null distribution for the test statistic (the sample mean) for the hypothesis that the sample is drawn at random.

You then see where the sample mean falls in the null distribution (specifically find the proportion of results at least as extreme as your sample proportion).

Here's an illustration for data somewhat similar to yours:

enter image description here

For a one-tailed test you compute the proportion whose means are at least as far from the population mean in the specified direction as your sample is (that's what's calculated above). If your alternative is two-tailed, you also need to identify what you mean by 'at least as extreme in the other direction' (it might be 'the mean is at least as far below', or it might simply be that you're after a similarly extreme quantile in the other direction -- which would result in doubling the tail proportion). In the two-tailed case, it may be easier to begin by thinking about a suitable form of rejection rule and then adjust your rejection rule to the desired significance level.

In either case if the resulting p-value is less than your significance level, you would reject the null hypothesis of random selection from the population.

All of the foregoing assumes that the decision to compare means was not made on the basis of what you see in the sample, but was a comparison that you would want to make before the sample was looked at.

If you want a more general test than a comparison of means, you might look at a similar procedure based off a goodness of fit statistic (but such broadening of the alternatives being considered will usually bring with it a loss of power). If the decision to restrict the comparison to a test of means was based on looking at the data (but you'd still have wanted some comparison), then this option may be about the only reasonable choice.

Glen_b
  • 282,281
  • Excellent. Is there a proper term for this kind of analysis? Monte Carlo something something? For what it's worth, this paper suggests that the proper p-value when empirically calculated like this is actually (r+1)/(n+1) where n is the number of replicates and r is the number of replicates in which the test statistic meets or exceeds that for the actual data. – Mike B Jun 03 '15 at 14:08
  • Well, in practice you almost never have the exact population to sample from, so I don't know that it does have a specific name, but this calculation (or rather, one based on all $500 \choose 11$ possible subsamples of size 11, which this can approximate as closely as you want) is what a large body of statistical theory tries to get at. You might call it a 'population-based randomization test' or you might call it an 'exact test of means with simulated p-value' or any number of similar descriptions. You could instead call it a "Monte Carlo exact test of means". – Glen_b Jun 04 '15 at 01:10