While studying elastic lasso, I have had a thought if I can apply a Bayesian method to the Elastic Lasso. If I want apply Bsyesian way of making a Regression model with Elastic Lasso, what do I need to think of?
Asked
Active
Viewed 79 times
1 Answers
4
In Bayesian models regularization is achieved by the choice of priors. For elastic lasso regression you need Bayesian linear regression with appropriate priors. To achieve $\ell_1$ regularization, you would use the Laplace prior, for $\ell_2$ regularization, the Gaussian prior. Elastic net means combining both regularization terms. As noticed by Lin and Lin (2010), this translates to the following prior
$$ \pi(\boldsymbol\beta) \propto \exp\left\{ -\lambda_1 \| \boldsymbol\beta \|_1 - \lambda_2 \| \boldsymbol\beta \|_2^2 \right\} $$
You don't need to normalize it for optimization, or MCMC sampling, so the only thing you need to do is to use the custom prior with the software you're already using.
However keep in mind that while those priors should technically lead to same results as in non-Bayesian flavor of the model, in practice there may be better choices for priors to achieve sparsity.
Tim
- 138,066
-
To use both regularization approaches, how do set the priors for both? – xabzakabecd Nov 11 '21 at 06:54
-
@StoryMay for both what? You need to use the prior mentioned above. – Tim Nov 11 '21 at 06:59
-
If I have 5 variables, do I need 10 priors for 5 Gaussian priors and 5 Laplace priors? – xabzakabecd Nov 11 '21 at 07:10
-
@StoryMay no you need single prior, mentioned above, for the vector of parameters. – Tim Nov 11 '21 at 07:20
-
Do you mean that I choose one prior for the vector of parameters? If possible, can you show me a small example or some further explanation in your answer? – xabzakabecd Nov 11 '21 at 08:25
-
@StoryMay it works same as any other Bayesian model, you multiply prior by likelihood and maximize the whole term, or sample from the posterior. – Tim Nov 11 '21 at 08:40
-
For example, here you have example how to do it in Stan https://mc-stan.org/docs/2_18/stan-users-guide/examples.html – Tim Nov 11 '21 at 09:25
-
I will look into it. Cherrs! – xabzakabecd Nov 11 '21 at 09:31