6

I am performing a meta-analysis. I have a subgroup made only of 2 studies and they show opposite results. The 1st has excellent outcomes the 2nd very bad ones.

Is it true that I am not allowed to compute the data from these two (different) studies under a random-effect model? I have been told that it is methodologically not correct to perform a meta-analysis of 2 studies with such opposite results.

They suggested me to report the raw data from these studies separately (without an overall outcome calculated by a random-effect model). Any suggestion? Or any place where I can find the explanation about this point?

mdewey
  • 17,806
Gabriele
  • 101

3 Answers3

10

I am not an expert, but I think it is common sense to not merge two studies if they have strongly opposite outcomes. For the sake of the example, say they measure a variable $X$ (glucose in the blood etc.) in identical conditions.

Since the outcome is different, you can imagine that there is "something" unknown, so that one of the team measures actually $X + a$. Now does that make sense to merge the records? You end up with a mixture model which is much more difficult to analyze. To give you an example, the mean is meaningless because it will depend mostly on the ratio between the sample sizes.

Addition: Thank you @Michael Chernick for this fantastic quote in the comments below:

Man puts one foot in a bucket on fire and the other in a ice bucket. On average the temperature is normal. You wouldn't want to describe this with the sample mean

gui11aume
  • 14,703
  • 9
    I agree with gui11aume. Also when reaults are so different it may indicate different assumptions or some other subtlety that makes the studies different and not proper to combine. It could be looked at a little like the old joke about statistics. Man puts one foot in a bucket on fire and the other in a ice bucket. On average the temperature is normal. You wouldn't want to describe this with the sample mean. It is the same with the meta analysis averaging results from two studies with widely different results. – Michael R. Chernick May 28 '12 at 11:53
  • 2
    That is a great answer, @Michael! To the point and memorable. – whuber Jun 22 '12 at 13:45
  • Indeed!! @Michael Chernick, with your permission I will put it in the answer for visibility. This one is too good :-) – gui11aume Jun 22 '12 at 14:28
  • Ok with me gui11aume. – Michael R. Chernick Jun 22 '12 at 14:47
2

try to find the confounding variable.. see if you get any interactions or mabe run another test with different methodology (and more variables that may give you a clearer picture.

uri
  • 21
  • 1
    Hi Uri, these 2 studies are the results of 2 different centers which perform the same surgical procedure to correct the same illness. One group has basically excellent results, the other one very bad, so the confounding variable is difficult to find. I guess a variable would be that one group has much more experience than the other to perform this kind of surgical procedure.... – Gabriele May 29 '12 at 12:52
1

Generally, there can not be bimodal findings. The theoretical underpinnings must be considered. Moreover, two sets of outcomes can not be combined in this way. If you are keen to apply meta-analysis, the outcomes showing excellent results and outcomes showing bad results may be meta-anlysed separately. Be sure that number of outcomes (effect-sizes for a group of excellent (or bad) results is more than say, 15. If the effect-sizes are in d-form, you may follow one of the several available random-effects formulas (Hedges and Olkin 1985) for checking the variablity in effect-sizes of r (see Davar 2006).

  • 6
    Could you be more specific about what "Davar 2006" refers to? – whuber Jun 22 '12 at 13:44
  • davar(2006) is an article in Decision. –  Dec 22 '21 at 11:37
  • Surely you can provide a specific reference! – whuber Dec 22 '21 at 14:27
  • 1
    Sorry for ambiguity, Sir. I wanted to tell that some formulas postulated by (Hedges and Olkin,1985) have been reported in the article which I got published in journal - DECISION (Indian Institute of Management, Calcutta, India), 2006. My article - meta-analysis of sample correlatios: The validity of Hunter, Schmidt and Jackson(1982) framework (S.C. Davar) is available at EBSCO HOST Website –  Dec 22 '21 at 15:39