6

There is a blind test that asks participants to distinguish Coke and Pepsi. A participant will test 6 cups of drink and tell whether it was Coke or Pepsi. Assuming that the participant can tell the difference between them, though not perfect, if he judged that the first three were all Coke, then he would think it is much more likely that the remaining three would be Pepsi rather than Coke. Then the last three experiments cannot give precise information about the participant's ability to distinguish.

Such a problem can happen when the participant knows a priori that there would be equal numbers of Coke and Pepsi. Then the experimenter might think of flipping a coin six times to decide which cola is given to the participant. Then it is possible that all or most of 6 cups are Cokes. It may be problematic when the participant is good at telling Coke a Coke but not as good at telling Pepsi a Pepsi.

Is there any methods to solve this problem?

mdewey
  • 17,806
user67275
  • 1,097
  • I don't see that there is a problem to solve, frankly. It will just affect what distribution the samples come from. – Peter Flom Oct 30 '12 at 14:30
  • 3
    Well, it seems that with only two alternatives telling Coke a Coke translates to telling not a Coke not a Coke, ergo a Pepsi. No? With more than two alternatives deducing the remaining flavors becomes difficult pretty fast. Still an interesting question (+1) – miura Oct 30 '12 at 14:32
  • miura : What I really meant is the probability that the participant classify Coke as Coke might be different from the probability that he tells Pepsi a Pepsi. Then does it make sense? – user67275 Oct 30 '12 at 15:31
  • Aren't you just dealing with the difference between two designs, one based on sampling with replacement and one with sampling without replacement? In the second case, of course the answers to consecutive sub-tests aren't independent, because it was designed that way, but that design presents no difficulties unless you try to pretend one design is the other. – Glen_b Oct 30 '12 at 23:30
  • There seem to be several problems here. Which one do you want solved? When the participant knows a priori there are equal numbers; when they just suspect that or are prejudiced that way; or when everyone knows that the cokes were chosen at random? – Peter Ellis Oct 31 '12 at 19:43

2 Answers2

2

I do think it is a study design problem and a famous one in that some think RA Fisher did not actually realize it in making his famous Lady tasting cups of tea example and one that haunts clinical trials who try to prevent any unblinding of treatment assignment in clinical trials.

A solution suggested from what is done there (choosing block size randomly) is to first randomly choose the ratio of coke to pepsi and then chose the order. Do make the choice of all coke or all pepsi very very low but not zero and then try to get "there is a chance that all will be coke or pepsi” into the informed consent.

phaneron
  • 1,312
0

You could (truthfully) tell the participants that you will flip a coin each time you provide a soda and that the coin flip will determine P vs C. You can go on to explain to them, "If the last five were Coke (or Pepsi), Coke and Pepsi are equally likely on the next test." One problem is that some of your participants won't believe the explanation and will remain convinced that five heads in a row makes tails more likely on the next flip.

But you might want to carefully simulate how you will analyze the data once you get it because randomly scrambling three and three is not the same experiment as flipping a coin on each soda.

Emil Friedman
  • 981