How do Machine Learning methods employ regularization to reduce variance?

Question

I was reading a paper and came across this sentence:

"ML methods perform well by employing regularization to reduce variance and trading off regularization bias with overfitting in practice.:

I am wondering if anyone would have any insight into this sentence. I understand that regularization involves shrinkage of parameters, but I fail to see how shrinkage reduces variance and what it means to trade off the bias? Does anyone have any further insights here? Thanks.

Have a look at the bias-variance tradeoff – user20160 Feb 13 '18 at 03:09 — user20160, Feb 13 '18 at 03:09

score 1 · Accepted Answer · answered Feb 13 '18 at 02:55

1

The variance referred to here means the variance between different models. Like eg if you fit 5 different nets, from scratch, with different initialization weights, you'll get 5 sets of results on the final test data, or on the dev/validation data. The stronger the regularization, the more similar the results will be, at the expense of fitting the training data less well.

For example, if you use L2 regularization, at the extreme of infinite L2 regularization, the weights will be 0, the output will be 0 for all inputs, and the variance will be zero.

answered Feb 13 '18 at 02:55

Hugh Perkins

4,697

What do you mean by infinite L2 regularization? Is it when, say in a Lasso, $\lambda \to \infty$? – user321627 Feb 17 '18 at 19:48
Like, if you added $\infty \sqrt{ \sum_i w_i^2}$ to your loss function (which would cause your weights to drop immediately to zero) – Hugh Perkins Feb 17 '18 at 19:51
What is the $\infty$ symbol in front mean? – user321627 Feb 17 '18 at 19:52
So, you would never use infinite L2 regularization. But yes, I mean something like $\lim_{\lambda \rightarrow \infty} \lambda \sqrt{\sum_i w_i^2}$ – Hugh Perkins Feb 17 '18 at 19:53
(but note that Lasso is L1 regularization; L2 is termed 'ridge', (I think)) – Hugh Perkins Feb 17 '18 at 19:54

How do Machine Learning methods employ regularization to reduce variance?

1 Answers1

Linked