6

I have looked around and haven't been able to find an answer to this question. Seems like an easy question, so maybe it's just a matter of looking in the wrong place.

I have a data set representing performance of assets across a large region. The assets are broken up into subregions as well. I can calculate mean and standard deviation for the region, and for the subregions.

How can I test if a subregion's performance is significantly worse or better than the overall region?

Here's some of my data: 

Overall: mean: 40% standard deviation: 15.3% sample size: 74
Sub-region A: mean: 43% standard deviation: 8.7% sample size: 10

I have 9 subregions. I think the answer might be some kind of T-test, but I'm not sure exactly which one is appropriate. Is sub region A significantly better than the overall average? Note: Sub region A's data is included in the overall statistics.

Andy
  • 19,098
Kevin Presley
  • 71
  • 2
    This is usually handled with ANOVA (which is amply described and illustrated on this site, as well as elsewhere). The most likely complication, though, is that performance could be subject to spatial correlation. Another complication occurs when you have different amounts of data in the various regions (due perhaps to their size or other characteristics), leading to a small area estimation problem. – whuber Feb 17 '15 at 22:11
  • Thanks! I can see both of those being a problem here. I'll give that a shot! – Kevin Presley Feb 17 '15 at 22:14
  • how about hypergeometric test ? – Areza Feb 17 '15 at 22:25
  • Okay, so I've just watched a video on ANOVA and have a question: I want to compare a subset from a full set. Is this appropriate? Should I remove the subset from the full set and have two sets? --I want to determine if a sub group is significantly better (or worse) than the group as a whole. – Kevin Presley Feb 17 '15 at 22:45

2 Answers2

2

To compare a subset with a whole that includes the subset is the same as comparing it with the while excluding that subset -- if it differs from one it differs from the other, and vice-versa. However, it's easier to do the calculations when the sets being compared don't overlap; both algebraically (because if you include the subset you will have dependence) and practically (this is the way standard statistical models are all set up).

Glen_b
  • 282,281
  • 1
    I might be misunderstanding, but if you include the subset values in the whole, doesn't that make the whole more similar to the subset than if they were excluded? E.g., t.test(x=1:2, y=1:5) gives p = 0.15; t.test(x=1:2, y=3:5) gives p = 0.049. What am I missing? – rbatt Jul 13 '17 at 22:45
  • 1
    @rbatt you're treating the samples as if they were still independent (you called t.test both times, which assumes independence) when in the second case you have a particular kind of dependence. To do the test properly you have to redesign your test to account for the dependence you now have. Once you do, it turns out to give the same results as a test where you don't have overlap and assumes independence, except it's more complicated to work all the details out. – Glen_b Jul 13 '17 at 23:56
  • 1
    Thanks for the clarification regarding the consequences of dependence! Would you happen to know a more formal term for this particular data structure (subset / whole)? Any detailed informal reference (e.g., Wiki article), or more formal citation (e.g., text book) that one might use regarding this concept/ procedure you've outlined? This seems like it would be a common scenario, though this is the first time I've encountered it in my data (I'm working on a problem that seems similar to the OP's). Thanks again. – rbatt Jul 14 '17 at 13:38
  • 1
    I'm not aware of a specific name nor of any reference for the procedure; it's reasonably easy to figure out; you end up calculating something that corresponds to the ordinary t-statistic with the overlap removed from the whole. The advice to just do the ordinary t-test with samples that don't overlap is in fact the simple way to get to the same place. – Glen_b Jul 16 '17 at 14:05
1

Another issue worth looking into when testing the same hypothesis on many samples (here, subsets) is that of multiple testing. Essentially, you want to avoid to reject the null too often when testing each hypothesis at (say) the 5% level, because that so to speak gives the data too many chances of producing a type-I error.

Christoph Hanck
  • 33,180