How does an ideal prior distribution needs a probability mass on zero to reduce variance, and have fat tails to reduce bias?

Question

I am reading this article about the horseshoe prior and how it is better than lasso and ridge priors. The author makes several points that I don't understand. One of them is "The ideal prior distribution will put a probability mass on zero to reduce variance, and have fat tails to reduce bias". Another is "both double-exponential and normal distributions have thin tails, and the probability mass they put at 0 is 0" How does the normal distribution have a zero probability mass at zero?. He goes on to explain how this makes the horseshoe prior better than LASSO and RIDGE priors. Yet, I don't understand why is this so and how does the probability mass at zero and at the tails improves the bias that occurs in the LASSO distribution.

My main question is ..what is the problem in LASSO and RIDGE priors that the horseshoe prior solves?

Answer 1

The MAP estimator can have non-zero probability mass at a point (even if the posterior distribution is always continuous)

The linked article is actually a bit misleading on this point, since even under the stipulated model all the relevant distributions are still continuous, so there is still zero probability mass at the point $\beta=0$. This is typical in penalised regression models, so it is misleading to describe things in the way the author has done. The issue here is really about the difference between properties of a posterior distribution, versus properties of a point estimator formed by taking the posterior mode (called the MAP estimator).

In regard to your first query, you should note that probability mass refers to actual probability ---not probability density--- so if a random variable has any continuous distribution then it has zero probability mass at any single point. This is true of the normal distribution, just as with other continuous distributions. The stipulated model in the linked post also uses a continuous prior distribution for the coefficient parameter in the regression.

The real issue here (which is obscured by the misleading language of the linked post) is that the estimator $\hat{\beta}$ used in penalised regression analysis can have a non-zero probability of being zero even when you use a continuous prior distribution for the true coefficient parameter. To see this, we first note that the estimator is obtained by maximising the log-likelihood plus penalty, which is equivalent to a MAP estimator (see this related answer for how these two approaches link to each other). Under certain specifications of the penalty function (equivalently the prior in Bayesian analysis) the MAP estimator has a non-zero probability of being equal to zero. In other words, it is possible to have:

$$\mathbb{P}(\beta=0) = 0 \quad \quad \quad \quad \quad \mathbb{P}(\hat{\beta} = 0) > 0.$$

This possibility may seem a bit subtle and it requires some explanation. A continuous prior leads to a continuous posterior in the regression analysis, so there is zero probability mass a posteriori at any given point in the parameter space. However, although every parameter value has zero probability mass a posteriori, the mode of the posterior (which is used as the point estimator) may be the same under a wide enough set of sampling outcomes that it has a non-zero probability of falling at a given point. This is a common occurrence in penalised regression analysis.

Answer 2

The idea is that you want your regularisation procedure to set small parameter estimates to zero and leave large estimates unchanged. Now, lasso does zero out small estimates (ridge doesn't even do that), but both lasso and ridge shrink large estimates towards zero, which is a significant source of bias in the two procedures.

For some intuition about why fat-tailed priors tend to leave large values relatively untouched, consider the ultimate fat-tailed prior, which is the improper flat pior $\pi(\beta) \propto 1$. In this case, the regression estimates are usual least-squares estimates and so are completely unbiased.

As for the probability mass question, both the normal and Laplace/double-exponential distributions have zero probability mass at zero in the sense that $\Pr_\pi(\beta = 0) = 0$. The advantage of having non-zero prior mass at zero $\Pr_\pi(\beta = 0) = p > 0$ is that this allows the posterior distribution of $\beta$ to have a positive probability of being zero, and so the posterior estimates of the regression coefficients are likely to have zeroed-out components. This reduces variance, since very small coefficient estimates are likely to be just be fitted to noise. Again, for intuition, we consider the limiting case, where $p = 1$ and so $\pi$ puts all probability mass at $0$. Now, the posterior estimate is always $0$, and so has zero variance.

Answer 3

Probability mass at zero

How does the normal distribution have a zero probability mass at zero?

The normal distribution has a non zero density at zero but the probability (mass) is zero $P[X=0] = 0$.

By placing a probability mass at zero the prior is expressing more strongly the believe that a parameter is probably zero. That is helpful in a setting where many regressors are included, of which we believe most are not truly in the model.

Fatter tails

By having fatter tails we allow for a few of the parameters to have more easily larger values. In RIDGE and LASSO the penalty is not only keeping out the 'unwanted' overfitting of noise due to too many parameters, it also makes that the 'correct' model parameters are shrunken. The estimated parameter values with penalization are smaller than with the unbiased ordinary least squares estimate.

What's the difference and which is better?

So you can see this prior as a more extreme variant of LASSO in comparison to ridge, placing even more focus on parameter selection and less on regularising by shrinking parameters.

Note that the one is not neccesarily better than the other. Shrinkage is not always unwanted and regularisation is not all about parameter selection. They are just placing a different focus.

The horseshoe

The name “horseshoe” came from the shape of the distribution if we re-parametrize it using this transformation using shrinkage weight k:

What is this 'shrinkage weight'? It relates to the use of the following prior model:

$$\begin{array}{} \beta_i &\sim& N(0,\tau \lambda_i) \\ \tau &\sim& \text{Half-Cauchy}(0,\tau_0) \\ \lambda_i &\sim& \text{Half-Cauchy}(0,1) \end{array}$$

or in reparameterized form

$$\begin{array}{} \beta_i &\sim& N\left(0,\frac{\tau}{\sqrt{\kappa_i^{-1}-1}} \right) \\ \tau &\sim& \text{Half-Cauchy}(0,\tau_0) \\ \kappa_i &\sim& \text{Beta}\left(\frac{1}{2},\frac{1}{2}\right) \end{array}$$

The relationship between the beta distribution and the half-cauchy distribution (whose square is F distributed), can be seen when we rewrite the reparameterization as $\lambda^2 = \frac{1-\kappa}{\kappa}$, and that resembles the transformation between the F-distribution and the beta distribution written in several places (e.g. Wikipedia here).

This $\kappa_i$ relates to the size of the prior $N\left(0,\frac{\tau}{{\kappa_i^{-1}-1}} \right)$ and makes it either a point mass when $\kappa = 1$ or a heavy tailed distribution when $\kappa = 0$.