While reading through KNN in detail, I was checking if there is a way to improve/penalize KNN? I didn't find any concrete/easily understandable solutions.
Asked
Active
Viewed 1,177 times
2
-
What do you mean by "improve"? – Sextus Empiricus Apr 24 '22 at 09:31
-
One way of regularization that I can imagine is in the selection and weighing of the parameters that are used in the model. – Sextus Empiricus Apr 24 '22 at 09:32
1 Answers
2
Good question. There are many forms of regularization: $\ell_1$, $\ell_2$, early stopping, dropout, etc. $k$NN is a nonparametric model, it doesn't have parameters to penalize, drop, or way to stop training earlier.
The main point of regularization is preventing overfitting. The way to prevent overfitting in $k$NN is to increase $k$ as it leads to averaging over many points instead of memorizing the training set as in $k=1$. It also makes the result smoother, because of averaging, which is another common consequence of regularization. A similar approach is used in decision trees where we restrict depth, minimal node size, or prune them, as means of regularization.
You can also use penalized distance metrics and in $k$NN regression more robust aggregation function, e.g. median, to make it less susceptible to outliers.
Tim
- 138,066