5

I have a simple question regarding the use of paired t-test and two-sample t-test:

  1. What is the consequence of using the independent two-sample t-test for dependent paired samples?
  2. What is the consequence of using the paired t-test for independent samples?

Thanks!

alittleboy
  • 1,013

2 Answers2

8

It would be bad -- very bad. If you ignore the pairing and use a two-sample t-test where you should have used the paired, chances are that you will not be able to detect the effect of interest. In this case, between subject variation is included in the estimate of variance used to measure the effect of interest. The variance is inflated, and only a substantial effect size will be deemed significant. Example: students are given a math test, followed by an instruction module, and then another test. If you ignore the pairing (before and after for each student), difference in ability is added to the effect of the module, thus making the effect harder to detect.

On the other hand, if you foist pairing onto an unpaired sample you reduce the degrees of freedom of the test. You are adding variables to the model (the pairing effect) that are basically random. In a sense, for each pair, you are adding a parameter for that pair -- but these parameters mean nothing. You would be adding noise, in a sense.

Placidia
  • 14,361
  • The former issue (ignoring pairing when it was actually appropriate, where one has substantially reduced power as per your answer) is probably less egregious (though still an issue) than the latter (adding in pairing where none exists.) I think the biggest issue with adding in pairing is that the (arbitrary) specification of pairs will have an influence on your estimates of differences between groups (including hypothesis test results). So you could have a biased result (for any one pairing choice), rather than just a noisier estimate of a difference. – James Stanley Jan 10 '13 at 21:33
  • If you grouped randomly, I don't think there would be bias - in the sense that the mean of the estimated difference between pairs would be the true group difference. Of course, if one cherry picked the pairs on the basis of data that was part of the same experiment and correlated with the outcome, any amount of bias could result. – Placidia Jan 10 '13 at 21:39
  • 1
    I should have followed my cardinal rule of thinking about statistics and done a worked example before commenting :-) You're right that the estimate of the mean difference wouldn't be biased for any arbitrary selection of pairs -- the standard deviation of the differences (calculated across the differences by "pair") would fluctuate though, and hence one's confidence intervals/p-values will vary depending on the pair selections (how pronounced this will be depends on underlying data distributions in the two independent groups and sample size). – James Stanley Jan 11 '13 at 03:58
  • 1
    @Placidia I've put my illustration of that last point in an answer below to serve as an adjunct to your (very good) answer. – James Stanley Jan 14 '13 at 03:21
4

I thought I'd post R code here to illustrate the point that I made in the comments to Placidia's answer above (which I like, but I wanted to go into more detail on this particular point.) This is the working that went into that comment.

The following code imposes random pairings of observations from two independent samples. If we repeat this imposition of random pairs 1000 times, running a paired t-test each time, you'll see that there is considerable fluctuation in the observed p-value. This is arising from fluctuation of the estimated std error and then t-statistic, of course, but I'm just illustrating the p-values here. But note that this problem has implications for the construction of confidence intervals too, rather than just for hypothesis tests per se.

I selected the random seed below to illustrate what happens when the unpaired t-test is "significant" (i.e. the "correct" analysis for these data, arising as they do from independent samples) but there is nothing canonical about that choice.

## Code written in R 2.15.2
set.seed(2464858)
## set a and b to be random selections from
a <- rnorm(20, m = .5, s = 1)
b <- rnorm(20, m = 1, s = 1)

# "Appropriate" unpaired t-test
t.test(a, b, paired=F)
# should return: t = -2.2397, df = 37.458, p-value = 0.03113

## Now 1000 iterations for imposing "fake" pairs at random: 
## with apologies for inefficient looping of code.
paired.ps <- array(NA, 1000)

for (i in 1:1000){
b2 <-   sample(b, length(b))
t.result <- t.test(a, b2, paired=T)
paired.ps[i] <- t.result$p.value
}

## Summary and histogram of distribution of p-values returned.
summary(paired.ps)
## returns    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##         0.00106 0.02533 0.03876 0.03936 0.05106 0.10000 

hist(paired.ps)

enter image description here

Note that you can tweak mean/standard deviation in the 2 populations below to see the impact of these parameters on the fluctuation of p-values from an (inappropriately applied) paired t-test.

James Stanley
  • 2,736