likelihood vs confidence in layman terms

Question

I already referred this post post and post. Don't mark it as duplicate.

I am working on a binary classification problem using logistic regression. Loan default or not.

I have a requirement where I am told to predict the outcome classes for our unseen records and also report the confidence of predictions generated.

I am novice data scientist. So, am not really sure what is the difference between likelihood and confidence?

If I run logistic regression, I get likelihood measure like 70% probability of belonging to class 1 and 30% of probability of belonging to class 0.

My questions are as follows

a) Does likelihood and confidence mean the same? Is there any simple explanation that ordinary layman like me can understand.

b) Is there any tutorial that you can suggest which has on how to report confidence of predictions? When I use scikit-learn for logistic regression, I don't know how can I report confidence for the predictions?

c) Any idea on how can we generate the interval? In scikit-learn logistic regression tutorials that I find online, I only see probability/likelihood of an instance becoming label 1 or label 0. Can you guide me on how can we generate interval?

Tomasz Bartkowiak · Accepted Answer · 2022-01-18T12:42:06.950

3

My guess is that Bayesian (Logistic) Regression is what you're looking for.

In a regular Logistic Regression you find the (model) parameters $\beta$ that maximise the likelihood of your data (see Maximum Likelihood estimation). In a binary classification you use those parameters to calculate the probability $p$ of a given (test) sample $X_{test}$ of belonging to a positive class.

In Bayesian regression, on the other hand, you're not trying to find single best parameters $\beta$ that fit your data. Instead, you're trying to find the distribution of $\beta$s given your prior belief about the parameters (e.g. you can assume that they're coming from a normal distribution centered at $0$ with some standard deviation $\sigma$*) and using your data to update your prior belief, arriving at a posterior distribution of $\beta$s. You can use the means of those posterior distributions to do predictions - this should give you results similar (or equivalent**) to predictions done using estimates from a normal logistic regression. However, having an entire distribution of $\beta$s (as opposed to a having a single value) gives you an idea about the variability of those estimates. A posterior distribution with high variance indicates that your estimate of $\beta$ has lots of variability, i.e. given a slightly different training dataset, the MLE estimate of $\beta$ can change a lot. You can use the standard deviation of the posterior distribution to calculate the confidence interval for the distribution of $\beta$. Now instead of reporting $\beta = 0.1$ you can say that e.g. 95% confidence interval for $\beta$ is e.g. $[0.05, 0.15]$. I believe that this is what your requirements refer to as confidence.

Remark You can perform predictions using both $\beta$s corresponding to the lower and upper bound of the confidence interval and this will give you two distinct predictions).

Here is a Python example of Bayesian Logistic Regression performed using PyMC3 package.

* What consitutes a good prior is a topic for a separate debate. Gaussian distribution centered at 0 is a good default choice. It also has a regularization effect as it forces the parameters to be close to 0.

** Logistic Regression with uniform prior will give results equivalent to MLE estimates.

edited Jan 18 '22 at 12:42

answered Jan 18 '22 at 07:56

Tomasz Bartkowiak

1,579

1

thanks for the help. upvoted – The Great Jan 18 '22 at 10:09
the machine always predicts “yes” with a probability between 0 and 1. So, is that called as our confidence score? – The Great Jan 18 '22 at 10:13
I found that from here - https://medium.com/voice-tech-global/machine-learning-confidence-scores-all-you-need-to-know-as-a-conversation-designer-8babd39caae7 – The Great Jan 18 '22 at 10:15
If Likelihood is same as confidence score of our predictions, then that is what I require. But is it same or different? – The Great Jan 18 '22 at 10:16
If you have time, can help me with this related question - https://stats.stackexchange.com/questions/560895/how-to-handle-hierarchical-categorical-variables-retain-or-drop – The Great Jan 18 '22 at 10:34
1

The confidence (interval) in this case would refer to $\beta$s, not the predictions themselves (at least not directly). Regular logistic regression gives you only one (MLE) estimate of $\beta$. Using this $\beta$ you can make a prediction, e.g. say that for $X_{test}$ it says that the probability of class 1 is 60%. In Bayesian Logistic Regression you get a range of $\beta$s (corresponding to the 95% confidence interval). If you use those two extreme $\beta$s to do predictions on the same test sample you'll get a range of probabilities of belonging to the class 1 (e.g. 55% - 65%). – Tomasz Bartkowiak Jan 18 '22 at 10:36
Yes, understand your point on CI. But does confidence Interval and confidence mean the same? – The Great Jan 18 '22 at 10:38
2

@TheGreat It is on you to define what confidence means. From what you wrote in your comments to me, it seems like you are interested in model validation to show a colloquial “confidence” that the model is a good one, rather than formal confidence intervals. – Dave Jan 18 '22 at 10:44
1

@TheGreat I don't know, although that would be my guess. Or it could be simply the difference between the $P(X = 1)$ and $P(X = 0)$, i.e. how confident your model predictions are for a given sample (i.e. 51%/49% is not confident, 99%/1% is confident). – Tomasz Bartkowiak Jan 18 '22 at 10:44
1

Note: As others have said you are incorrect in labeling this as a classification problem. The Bayesian approach is most suited for communicating uncertainties. Short of that, computing 0.95 confidence intervals for predicted probabilities for selected covariate settings will be helpful. This uses standard logistic regression methods, with a normal approximation for the predicted log odds of the outcome. – Frank Harrell Jan 18 '22 at 13:32

likelihood vs confidence in layman terms

1 Answers1