4

I've recently asked whether it is valid to treat repeated measures data as both within- and between-subject data and compare the analysis of both to see if there are differences. My question was motivated by a particular paper (PDF) about presence questionnaires that, in my opinion, implied that such questionnaires are essentially not applicable in between-subject designs, as people would only be able to reliably rate their sense of presence in a particular environment relative to another environment.

This assumption was questioned by some of the answers, and hence I was wondering whether treating the same data as between-subjects and within-subjects data can ever lead to a systematic difference. That is, is it possible that an effect will show up if I analyse data as within-subjects but not show up when analysed as between-subjects?

By way of a very contrived example, assume I get subjects to rate on a scale of 1 to 10 how much they like an offer of free beer. In one condition, people are offered one litre of beer for free, in the second condition, people are offered two litres of beer for free. If I ran this experiment with a between-subjects design, I would personally assume that there isn't any discernible difference between the two conditions, because - hey, free beer! But as a within-subjects design, I think I stand a reasonable chance of seeing an effect, because more free beer is better than less free beer.

Community
  • 1
ThomasH
  • 251

2 Answers2

3

I think you already mentioned one of the most important points, knowledge about the different conditions. If my response in one condition totally influences the response to another condition which can only be avoided in between-subjects designs, this may be especially problematic with insight problems but you could construct a myriad of different examples.

In cases where you might be unsure on whether it has an influence you could use a combination. Imagine you have two treatments that could be administered both between or within-subjects. Divide your sample into two (random) groups in which one group starts with treatment A followed by treatment B and the other with treatment B followed by treatment A. You can now compare treatment A and B using only the first measurement (i.e. between-subjects) and compare them within-subjects. If both analysis agree you have evidence that it is not the case for your task.

In general, I think you question is more suited for cognitive science stack.exchange as it is not statistical (i.e., I voted to close).

Community
  • 1
Henrik
  • 14,198
  • 11
  • 69
  • 130
  • Thanks for getting the idea behind my rambling post. It is indeed more a CogSci question (and duly posted), since I'm interested in situations where this happens (think carryover effects). Am I right then in assuming that, if I performed a between-analysis with greater power (through more subjects) than the within-design, and I get an effect in the latter but not the former, that my effect is due to the exposure to multiple treatments rather than an effect of the treatment itself? – ThomasH Nov 25 '12 at 23:59
  • @ThomasH In general, within-subjects design are usually way more powerful, hence effects there are not that easy to dismiss. You need to do a power analysis to show that the between-subjects analysis is really more powerful. (You will need the correlation of the within-subject measures). I would Use g Power 3 for that. – Henrik Nov 26 '12 at 09:03
1

Certainly it's possible an effect can be significant within S that's not significant when the repeated measures experiment is analyzed between S. The actual numerical effect (not standardized) would be the same in both cases though.

If there were no difference one might logically conclude that there was no effect of the analysis being within S and therefore the effect and subjects are not correlated. That would be incorrect because the difference between significant and not significant is not, in itself, significant. Furthermore, there could also just be really low inter-subject variability. You can't tell from the approach you're suggesting.

Such a difference in analysis within a study isn't really sufficient to motivate the experimental design. Within and between S design should come from logic, not from an analysis grounded in a poor understanding of the difference between analyzing repeated measures and between.

John
  • 22,838
  • 9
  • 53
  • 87