1

If I had a small number of observations (about 5) in a sample and the population standard deviation was unknown, what test could be done to see if the value of one of the observations was significantly different from the mean? An example of this would be to test if one judge out of 5 judges in a competition was giving a score significantly different from other judges.

gung - Reinstate Monica
  • 145,122
eglinker
  • 41
  • 1
    The answer is yes, but if you're using the same data to test it as you did to identify the judge you want to test, your p-values will be meaningless (unless they take account of that fact). That is, if you say "man, that judge is harsh" (say), then "let's test if he's scoring differently", you can't test the data that made you notice him. – Glen_b Jun 10 '14 at 00:35
  • 1
    Is your 'judge giving a different score' just an example, or is it the real question? There are methods for assessing inter-rater agreement. – gung - Reinstate Monica Jun 10 '14 at 00:37
  • 1
    There's an example here of a t-test with $n_1=3$ vs $n_2=1$, as an indication that (with some uncheckable assumptions) such things are possible -- but @gung's note about inter-rater agreement may be nearer to what you need (with the same caution about using the data that made you want to test). But it also seems to me that a single judge in a competition would presumably give a string of scores, not a single score, so this doesn't sound like an $n=1$ problem. Can you clarify? – Glen_b Jun 10 '14 at 00:41
  • @gung,the judge example is the real question. I am not trying to resolve a particular situation with a particular judge at this time, but I would like to be prepared just in case the occasion does occur. – eglinker Jun 10 '14 at 01:09
  • 1
    Will you really need a significance test? As @Glen_b notes, if you notice one judge giving different scores & decide to test that judge, the p-values will be meaningless. Will there be just one score, or will there be many scores you can use (eg, ratings of ice skaters by different judges)? You might also be interested in multivariate outlier detection. – gung - Reinstate Monica Jun 10 '14 at 01:19
  • @Gung, there would be for example, many scores (say 5) from different judges for the same skater. – eglinker Jun 10 '14 at 02:03
  • Originally I was hoping that I may be able to able to test a single score against the scores given by a rest of the judges in a single match. However, after viewing the ecology project example given by @Glen_b I am thinking I should be treating of a string of scores from one judge as one sample and the string of scores of all the other judges combined as the second sample. Then I should use student t to determine if there is a significant difference between the sample means. Would that be the correct interpretation? – eglinker Jun 10 '14 at 02:04
  • Right, but what I mean is would you have a profile of scores (say 50 scores for 50 different skaters) from each judge? That way, you would be better able to see if a judge is systematically lower (eg) or favors skaters of a particular nationality, etc. – gung - Reinstate Monica Jun 10 '14 at 02:05
  • If the judges are judging people of different ability, you may need to deal with that, as well. If you want to extend your inference outside that particular competition, you could treat them as random effects. If you the competition is the population you could treat the skaters as blocks (covariates, fixed effects). – Glen_b Jun 10 '14 at 02:32
  • @gung, Originally I hoped to use 1 score per judge for 1 skater (5 judges), but now I believe that 5 to 50 scores for 5 - 50 skaters is the way to go, that way I have the sample standard deviations for the both samples (the scores from one judge and the scores from rest of the judges). – eglinker Jun 10 '14 at 02:37

0 Answers0