3

Many psychology studies draw conclusions from experiments that follow a design along the lines of:

  • take two groups, and subject each of them to a different manipulation (e.g. task instructions etc)
  • measure both groups on one and the same dependent variable
  • if a difference is found in that DV, claim that the manipulation is what explains it.

For instance, take two studies done by D. Kahneman et al. (don't have exact refs sorry):

  • Study 1: both groups asked to read sentences and either agree or disagree with their truth value; group A is asked to nod their heads (yes) while reading, while group B is asked to shake their heads (no). Group A is found to agree with the statements in a higher proportion than group B.

  • Study 2: both groups are asked to make liking ratings on topics T1 and T2, but the ratings are presented in different orders (group A: T1 then T2; group B: T2 then T1). A correlation between ratings of T1 and T2 is found for group B but not for group A.

Is this type of inference well-founded? To me it seems not - unless there is additional evidence of the two groups not differing in the absence of (or prior to) the different manipulation, i.e. some form of proof for a baseline equivalence in the absence of the experimental manipulation. Such evidence is, however, almost always missing from the studies reporting them, with the few exceptions of those reporting "pre" and "post" measurements (which satisfy the condition I mention above).

In the absence of such additional evidence, how is the trivial confound of the two groups simply having different baseline preferences for the thing they are require to rate, a confound that could well explain the results single-handedly, meant to be corrected for? Surely a simple independent samples t-test is not enough to account for these baseline differences, but is it just a case of replacing this with a more sophisticated statistical test?!

z8080
  • 2,370
  • 1
    +1. The elephant in the room that you are missing is random assignment of subjects to groups. – amoeba Sep 18 '16 at 12:40
  • Good point - I should have mentioned that, despite assuming this random assignment has been properly done, without a baseline comparison I still find it hard to attribute any effect solely to the manipulation, when it could well be that, even without it, the groups would have differed anyway - they just weren't tested before the manipulation, only after it. – z8080 Sep 18 '16 at 13:42
  • One more thing: random assignment of subjects to groups should really not make one expect that the groups would be the same on any measure, whether interfered with (manipulated) or not. If I take a bunch of random people, measure their height and take the average, then do the same for another group, there's no reason to believe these two averages will be the same. – z8080 Sep 18 '16 at 22:57
  • So any more thoughts? :) – z8080 Sep 20 '16 at 14:14
  • 1
    You have $n$ subjects participating in the experiment. If you randomly assign $n/2$ to group A and $n/2$ to group B, you have all the reasons to expect that subjects in group A will have the same average height as subjects in group B. Provided of course that $n$ is not too small. But you are right: it's usually better to measure each subject pre and post and account for that. – amoeba Sep 20 '16 at 14:26

0 Answers0