4

A nice property of maximum likelihood estimators is that, while they can be biased, they are consistent for $iid$ observations.

In a logistic regression, unless the conditional distributions all have the same probability parameter, we lose the "identically" distributed of $iid$. Nonetheless, logistic regressions tend to have their parameters estimated by maximizing the likelihood.

(Why) does this not lead to an inconsistent maximum likelihood estimator?

Related post: Logistic regression panel data fixed effects.

User1865345
  • 8,202
Dave
  • 62,186
  • the MLE is consistent and asymptotically normal: this is the way that most software gets confidence intervals in GLMs. a popular reference for this topic Chapter 6 of Lehmann and Casella: http://dcpehvpm.org/E-Content/Stat/E%20L%20Lehaman.pdf – John Madden Jan 30 '23 at 23:05
  • 1
    That's what I thought, too, but evidently not. @JohnMadden – Dave Jan 30 '23 at 23:23
  • Oh fascinating let me take a moment to read through all that – John Madden Jan 30 '23 at 23:31
  • Oh, are you interested in the panel data case in particular, or standard logistic regression? If my (super briefly gained) understanding is correct, the issues from the linked question arise from the panel-specific parameters, right? – John Madden Jan 30 '23 at 23:32
  • @JohnMadden There, I only care about panel data in particular. Here, I care about a standard logistic regression (though maybe another question of mine or a tangent in the comments can be why a panel data regression is not a special case of a standard regression). – Dave Jan 30 '23 at 23:36
  • 1
    "i.i.d." is a sufficient condition, but not a necessary condition for consistency. – Zhanxiong Jan 31 '23 at 02:20
  • 1
    @Zhanxiong But why is there consistency despite the lack of $iid?$ – Dave Jan 31 '23 at 02:28
  • 2
    @Dave Whereas a rigorous discussion requires a careful listing of assumptions on the parameter space, a quick answer is that: the MLE is a type of $M$-estimate, and the log-likelihood function of the logistic regression (under the usual regularity conditions) satisfies the condition of the Wald's consistency theorem (see Theorem 5.14 in Asymptotic Statistcs by A. W. van der Vaart), thus $\hat{\beta}$ is consistent. – Zhanxiong Jan 31 '23 at 02:52
  • 1
    @Zhanxiong Interesting, thanks! But then what goes wrong in the panel data setting of my linked question? – Dave Jan 31 '23 at 02:58
  • 2
    Dave, I added your other question's link in this post for the commentary there is definitely relevant here. – User1865345 Jan 31 '23 at 04:40

1 Answers1

2

Some conditions on the explanatory variables are required for consistency

As with other regression models, the MLEs in logistic regression are not necessarily consistent --- consistency requires imposing some assumptions on the sequence of explanatory variables used in the limit. In the case of Gaussian linear regression we obtain consistency of the parameter estimator using the Grenander conditions (see related answers here, here and here), which (roughly speaking) requires the maximum leverage of the explanatory variables to converge to zero. It is likely that a similar requirement would be needed to prove consistency for the case of the logistic regression. In general, so long as the maximum leverage converges to zero you get a situation where no individual data point, or finite set of data points, is "influential" in the limit.

In view of this, the real question you need to be asking is: what is a reasonable set of sufficient conditions we can impose on the explanatory variables which yield a consistent MLE for the parameters in the logistic regression model? If you undertake this inquiry and find sufficient conditions of this kind, you will be able to see how consistency is established (usually in combination with asymptotic normality) and you will therefore be able to see why the required conditions are weaker than requiring the response variables in the model to be IID. While there are lots of different sufficient conditions that can be formulated, I recommend starting off by reviewing some standard conditions (see e.g., Fahrmeir and Kaufmann (1985) to get you started).

Ben
  • 124,856