9

Say I have a binary outcome of 0 or 1 and suppose I were to use logistic regression model to estimate the probability a new sample will have an outcome of 1.

I have read answers (for example here: Computing prediction intervals for logistic regression) that indicate it's nonsensical to compute prediction intervals, as the outcome can only be 0 or 1.

However, is there nevertheless a sensible way to compute a prediction interval in log-odds space, before transforming it using the logistic function into a probability interval?

My goal is to be able to communicate a level of uncertainty in each new prediction I make (e.g,. "This new product is estimated at a 0.40 probability of having the desired outcome, but the prediction interval around this estimate is [0.3, 0.5]").

Community
  • 1
Carl S
  • 361
  • 1
    maybe I'm confused, but aren't you just looking for a confidence interval rather than a prediction interval here ??? – Ben Bolker May 01 '15 at 21:12
  • 1
    Prediction intervals for the response would seem to be be pretty useless. Where you have modeled >95% in one of the two categories, your interval will be width 0, and everywhere else it will be width 1 (and a 100% interval). The boundary where that changes to the wider interval carries some information, but from a human-point of view it's not really telling us what a prediction interval does when dealing with continuous data or even discrete data that's more "spread out" over a greater number of likely values (like say a Poisson with moderately large mean) – Glen_b May 02 '15 at 03:32
  • 2
    I think you bring up an interesting point regarding how the shape of the underlying distribution affects your prediction interval, but I'm not sure I understood the rest of your comment.

    To better illustrate what I'm interested in: I am curious about statistically sound principles behind calculating the standard error for predictions on a new datapoint, so that I can communicate uncertainty in my prediction.

    In this specific case of logistic regression, I'm wondering how this could be achieved. I've seen this done for OLS, and wonder if there are similar concepts for logistic regression.

     – Carl S May 02 '15 at 04:45

2 Answers2

2

In the log-odds space, every outcome is either $-\infty$ or $\infty$, so a prediction interval still makes no sense. You can achieve your goal by giving a confidence interval for the probability.

Brent Kerby
  • 2,463
  • 13
  • 11
  • 1
    In log-odds space, outcomes will be $-\infty$ or $\infty$ only after thresholding the probabilities produced by logistic regression to 0 or 1. I am interested in whether you could generate a prediction interval before thresholding.

    I was wondering whether there are statistically sound principles for giving a prediction interval for the predicted probability (before thresholding), perhaps by what Sebastian is suggesting above.

     – Carl S May 02 '15 at 04:32
2

That's a little bit confusing. If you are doing logistic regression, you predict binary class memberships, e.g., 0 and 1. The class label is basically the outcome of a thresholding function that is

$sigmoid(z) \ge 0.5 \rightarrow 1 $, and 0 otherwise.

where $z = w^Tx$ (w=weights, x=sample), and $sigmoid$ is the inverse-logit function $\frac{1}{1 + e^{-z}}$

or equivalently to save this one step of computation you can directly compute:

$z \ge 0 \rightarrow 1 $, and 0 otherwise.

The conditional probability $p = sigmoid(z)$ is basically your "confidence". I think what you want is the confidence of this probability? You could do this by calculating the standard error of your prediction on the linear scale $w^Tx$, and then calculate the upper and lower bounds of your, e.g., 95%, confidence interval via $[ pred. value +/- 1.96\times std. err ]$. After you obtained the upper and lower bounds, you can use the sigmoid function to transform those onto the logit scale.

  • Thanks Sebastian. Your interpretation of what I want (the confidence in probability) is correct, and your answer is essentially what I would probably try at first pass.

    Do you know of any papers/theory that dive more deeply into this issue? I am curious whether this solution has statistically sound principles behind it, as I have only seen discussions of prediction intervals around OLS, and not logistic regression.

     – Carl S May 02 '15 at 04:28
  • 2
    @Carl, just did a quick google search, and Section 1.4 in , Applied Logistic Regression by Hosmer and Stanley may be what you are looking for then! –  May 02 '15 at 06:07