1

I'm pretty new to Bayesian statistics so hopefully this makes sense:

I'm fitting a Bayesian ridge regression using scikit-learn whose target values lie between 0 and 1. When the point values (the mean of the predicted distribution) are compared to the observed (gold) point values, MAPE, WAPE, and r^2 look good. But when I generate predictions with the standard deviation of their distributions, the standard deviations are often large. For instance, I'll get a prediction like 0.01 -- good when compared to the expected value -- along with a standard deviation of 1.78.

Obviously, such a large standard deviation indicates a large amount of uncertainty around the predicted mean, and therefore more work needs to be done on the model to increase certainty. However, I had no indication of this since MAPE, WAPE, and r^2 on the predicted mean don't provide insight into this.

What other metrics should I be looking at here?

Possibly related:

dmn
  • 111
  • 1
    Re "... and therefore more work needs to be done...": that's not so evident. You should expect arbitrarily large SDs no matter how good the model might be when the explanatory values lie far from the data (in the Mahalanobis distance). Could you explain why you think changing the metric might help? – whuber Jun 28 '23 at 14:36
  • That makes sense. It's not so much changing the metric, just computing an additional one (or two or three) that conveys potential "problems" with the predicted standard deviations. But it's interesting that you bring up the Mahalanobis distance between the standard deviations and the features as the standard deviations all look to be in the same scaled units of their features... – dmn Jun 28 '23 at 14:56
  • Since you're doing ridge regression, you likely have some directions (in the space spanned by the explanatory variables) where the response varies rapidly. If the point at which you're making the prediction has any component in those directions, you can expect to have enormous prediction standard errors (unless you have regularized extremely aggressively). Simply examining the individual SDs won't tell whether that's the case. – whuber Jun 28 '23 at 17:00

0 Answers0