10

I was given an article reporting a study very similar to one my lab wishes to run. But, I noticed that for the variable of interest, Duration, the SDs are larger than the mean...since this is duration measured in minutes it can never be negative and this seems very strange to me. This happened in 2 studies reported, below is one.

Beyond that, this is a mixed-design. Control v Treatment (between groups), and Time1,Time2,Time3 (repeat measures). Here are the means (SDs), N >200 

                       Time1                Time2                  Time3 
Control               15.1 (14.6)          14.4 (14.8)            13.3 (15.7)
Treatment             14.8 (13.2)          10.0 (12.2)            8.2 (9.9)

...they ran an ANOVA and reported a p<.001.

I was asked to use this as a basis for a power analysis to determine sample size for our study. I am pretty sure this indicates that the data is non-normal or has outliers and I don't feel comfortable determining sample size based on this. Am I just way off base?

Glen_b
  • 282,281
Curious
  • 101
  • Are you sure those are SD and not 95% confidence intervals which might be more like 3*SD. It looks like those SD are about the same size as the means. It's really hard to say what the sample size is as we don't know what effects were included in those errors or even what the statistic is. If it were just counting statistics,i.e. the Poisson distribution, included the mean over the SD should be like 1/Sqrt(N). However that would imply N=1 (or a few at most). Can you give us more information about what these statistics are? – Dave31415 Mar 19 '14 at 21:05
  • Also, the normal distribution has a mean and SD that are completely independent of each other. I think perhaps you meant the Poisson distribution. – Dave31415 Mar 19 '14 at 21:05
  • 2
    With durations that are non-negative, I'd usually expect a skewed distribution. SDs comparable to the mean are perfectly possible and in no sense surprising. What distribution is best assumed for other calculations can't be advised without more information, but I would not choose Poisson as my first guess, but rather gamma or lognormal. – Nick Cox Mar 19 '14 at 21:23
  • As @NickCox notes, w/ durations I would be surprised if the SD wasn't larger than the mean (if there was no censoring). You might also consider the Weibull distribution. Power analysis will probably have to be simulation-based. On a different note, I would guess an ANOVA was invalid w/ data like that. – gung - Reinstate Monica Mar 19 '14 at 21:31
  • The N for the above study was 206, and these are the means and SDs as reported in a table in the paper. I was told to try to use this older paper to determine sample size for a new study. The variable is essentially the number of minutes each subject spent working on a problem. Seems as though I'll have to report that we can't do the power analysis without the raw data. – Curious Mar 19 '14 at 22:02
  • 1
    For a data set of $n$ nonnegative numbers, the coefficient of variation -- the ratio of the standard deviation to the mean -- can take on values as large as $O(\sqrt{n})$ with the maximum value occurring in the extreme case when all the numbers are $0$ except for one (see this question for details). Thus, the standard deviation exceeding the mean should not be regarded as an exceptional case requiring lots of explaining away. – Dilip Sarwate Mar 19 '14 at 23:04
  • When the SD approximates the mean across several groups, that is a clear indication that an analysis based on the logarithms may be warranted (and is likely to be more powerful). Although such an analysis would be comparing geometric group means rather than arithmetic means, if the objective is just to identify differences among groups then this should be an attractive solution. A rough power analysis can still be conducted (based only the paper's summary statistics) by assuming lognormal variation. – whuber Mar 20 '14 at 21:34

2 Answers2

5

It's easily possible for the standard deviation to exceed the mean with non-negative or strictly positive data

I'd describe the case for your data as the standard deviation being close to the mean (not every value is larger and the ones that are larger are generally close). For non-negative data, it does pretty clearly indicate that the data are skew (for example, the gamma distribution with coefficient of variation =1 would be the exponential distribution, so if the data were gamma, they'd look somewhere near exponential)

However, with that sort of sample size, the ANOVA may not be particularly badly affected by that; the uncertainty in the estimate of pooled variance will be pretty small, so we might consider that between the CLT (for the means) and Slutsky's theorem (for the variance estimate on the denominator), an ANOVA will probably work reasonably well, since you'll have an asymptotic chi-square, for which the ANOVA-F with its large denominator-degrees-of-freedom will be a good approximation. (i.e. it should have reasonable level-robustness, and since the means are not so very far from constant, the power shouldn't be too badly impacted by the heteroskedasticity)

That said, if your study will have a smaller sample size, you may be better off looking at using a different test (perhaps a permutation test, or one more suitable for skewed data perhaps one based on a GLM). The change in test may require a somewhat larger sample size than you'd get for a straight ANOVA.

With the original data you could do a power analysis under a suitable model/analysis. Even in the absence of the original data, one could make more plausible assumptions about the distribution (perhaps a variety of them) and investigate the entire power curve (or, more simply, just the type I error rate and the power at whatever effect size is of interest). A variety of reasonable assumptions could be used, which gives some idea of what power may be achieved under plausible circumstances, and how much larger the sample size might need to be.

Glen_b
  • 282,281
4

You are correct in concluding that the data is non-normal. If the data were normal then we would expect about 16% of observations to be less than the mean minus the standard deviation. With an SD larger than the mean this number is negative and you state that there cannot be negative numbers, so what you are seeing is not consistent with normally distributed data. The SD values are possible, but only if the distribution is very right skewed (which is common in durations).

I agree that choosing a sample size based on assuming data will be normal is not a good idea, but if you can find out more about the process and find a right skewed distribution (a gamma distribution as one possibility) that is a reasonable assumption, then you could use that to help determine sample size.

Greg Snow
  • 51,722