6

When using a Poisson GLM, its dispersion parameter can be estimated as "residual deviance/degrees of freedom".

But when analyzing Poisson data with a Poisson model (i.e., not overdispersed), I found that this estimated dispersion is not around 1 based on simulation. The deviation from 1 is strongly influenced by the sample size and the mean of the Poisson distribution. Because I was expecting it to be distributed around 1, I am confused.

nsim <- 1000
disp <- rep(NA,nsim)
n <- 1000

for(i in 1:nsim){
disp[i] <- glm(rpois(n,1)~1,family=poisson)$deviance/(n-1)
}


hist(disp)

enter image description here

quibble
  • 1,436

1 Answers1

8

That's correct!

You've found out why glm doesn't use deviance/df as an estimate of dispersion: it's not a very good one. It uses the better estimate based on the variance of the Pearson residuals (though for family=poisson it doesn't need to estimate).

The estimate is bad because the deviance residuals aren't actually that close to $N(0,1)$ under the model -- they can't be, because $Poisson(1)$ is discrete, and $N(0,1)$ isn't. However, the Pearson residuals do have variance very close to 1 (exactly 1 if you didn't need to estimate the mean), and so give a better estimate of the dispersion.

> r<-replicate(1000,glm(rpois(n,1)~1,family=poisson)$deviance/(n-1))
> mean(r)
[1] 1.146889
> s<-replicate(1000,summary(glm(rpois(n,1)~1,family=quasipoisson))$dispersion)
> mean(s)
[1] 1.000165

For the Poisson family, the bias looks like this with varying mean. The blue is the deviance-based estimate; the orange is the estimate based on Pearson residuals enter image description here

This is a related phenomenon to $\chi^2$ tests in contingency tables not being very accurate with small cell counts -- again, the $\chi^2$ approximation is based on Poisson distributions being approximately Normal, and when the mean is 1 they aren't.

Note, however, that a bit of perspective is useful here. The dispersion estimate based on the deviance is biased when the mean is 1, but it's not all that biased. If you want to know the dispersion to within 10%, you need large sample sizes and you need to know the distribution is accurately Poisson. The bias in the estimator probably isn't your biggest problem.

Thomas Lumley
  • 38,062