4

I'm comparing two groups A, B after cross tabulating them with cases and controls.

I get a table as such:

    Control Cases
  A   8       0
  B  14       0

Obviously, I can't do an odds ratio because of the 0's. Doing a Fisher's exact test gives me a p-value of 0.5152 but the confidence interval goes from 0.16 to Infinity.

Adding .5 to each cell doesn't change Fisher's exact test (although I'm not sure why the ODD's ratio is still infinite in R) and I'm hesitant to use a normality approximation due to the small cell counts.

However, the signal seems to be quite strong that A and B both act the same... Neither of them are push an observation into the "case" category. (Here A & B are toxic substances).

What would be some other ways I can tackle this to get this point across?

Thank you!

Fomite
  • 23,134
user1357015
  • 1,744
  • 1
    Your terminology is likely to confuse people: ordinarily "controls" and "cases" are assigned by the experimenter. If you have no cases, you haven't done the experiment yet and there is nothing to analyze. One might guess that "controls" and "cases" just distinguish two possible outcomes that can occur, but what exactly do "A" and "B" mean? Were they determined by the experimenter or are they observations of some binary outcome, too? – whuber Apr 23 '14 at 18:57
  • A & B are two toxic compounds. Cases and & Controls were if biological damage was observed after administration of either A or B. – user1357015 Apr 23 '14 at 20:07
  • 1
    @whuber This is not necessarily true. Consider the canonical Case-Control study, wherein cases and controls are not assigned by the experimenter. In many circumstances, these studies also struggle with low cell-counts. – Fomite Apr 23 '14 at 22:45

2 Answers2

4

In this case, my best recommendations are as follows:

  1. Think about your experiment
  2. Get more samples

The first is to address a possibility within your data - that you're running into 0 counts for the cases because the treatments you're using are incapable of producing cases. That is, you don't have 0 cases due to random chance, but you have 0 cases because p(Case|Treatment) = 0. Do you have any support for the belief that A or B can cause biological damage at the levels you are administering them?

If you don't, this may be a pathological problem that cannot be fixed purely with statistics, and will require revisiting your study protocol.

The second is, well, that if A or B can rarely cause biological damage, you may be making valid inferences using Fisher's Exact Test or the other ways to deal with small cell size, but because both treatments have zero cases, you're going to get wildly imprecise estimates, as seen from your confidence intervals. In this case, your study is simply under-powered, and you need more samples.

Both are experimental design suggestions, because at this point what your asking is for statistics to show that two things that aren't different are different. That's a tall order.

Fomite
  • 23,134
1

A solution could be using a bayesian approach, by putting a weakly informative prior on the estimates; specifically, a prior which gives very low probability to extremely positive or negative values to the log of the Odds Ratio

(remember: Odds Ratio [0; Inf] -> log Odds Ratio [-Inf; Inf])

.

Without going full Bayesian, I often use a semi-Bayesian (aka penalized maximum likelihood estimate) estimation for the log odds ratio, imposing a Cauchy prior (see Gelman http://www.stat.columbia.edu/~gelman/research/published/priors11.pdf), which is actually a T distribution with df = 1.

Here's what I would do in R (with arm library):

mod <- bayesglm(cbind(c(0,0), c(8, 14)) ~ c('A', 'B'), binomial()) # B versus A
exp(coef(mod)[2]) # OR ≈ 0.95
exp(confint.default(mod)[2,]) # CI ≈ 0.011, 81.4

Of course, the analysis shows the total lack of information in the data regarding the phenomenon; it just put real numbers to uncertainty instead of infinite values, but nonetheless a quite uninformative interval!

In summary, the interpretation is that this data cannot tell you whether A is different from B. You need more data and you need to observe at least few cases! Than Bayes or semi-Bayes methods can help you with lack of stability in the case or rare events by shrinking estimates to more credible values.

Bakaburg
  • 2,917