2

considered KNN, it seems that a machine learning algorithm does not have to have weights, a training process, loss function or optimization, so, what is the common ingredients of a machine learning algorithm?

it seems that what a machine learning algorithm does have to have includes a training set, prediction functionality, distance metric, what else?

A lot of machine learning books/tutorials use KNN as the 1st example of supervised machine learning without telling the keys we can get from KNN, So, I am trying to figure it out.

JJJohn
  • 1,875
  • 2
  • 12
  • 28

1 Answers1

0

You might want to check the definition of "supervised" & "unsupervised" statistical (machine) learning. And most importantly check & understand the methods you want to apply. I wouldn't generalize "machine learning algorithm".

For your example. KNN is an extremely simple nonparametric method. It doesn't have to be trained. There is no parameter!. It just checks the k-nearest neighbor.

You clearly have to find the best "k" though. So you can test different k, calculate e.g. RSS and chose the best k. Guess you can define that as a loss-function :-)

Clemens Haerder
  • 86
  • 1
    I would argue that a KNN classifier definitely has parameters, in fact an unbounded number: it needs to save all of its training data. – Mees de Vries Aug 03 '19 at 08:44
  • Does the training change "saving the training data"? The data is known, hence not a parameter to be estimated. I would argue that this is just required to run KNN. But there is no training involved but computational resources. It is nonparametric per definition I would say. The only unknown hyperparameter is k. And finding a reasonable one is described above. – Clemens Haerder Aug 03 '19 at 08:55
  • Thanks for your answer. In the context of supervised machine learning, What do KNN and other supervised machine learning algorithms both have in common? – JJJohn Aug 03 '19 at 11:45
  • I would say the task :-). Classification or regression (categorical or numerical response). For KNN it doesnt make sense to split your data in a training, test, validation set as it just decreases the performance. As said, there is no training required as kNN is nonparametric. But I am not sure if this answer helps, or if there is a good answer to that question... – Clemens Haerder Aug 04 '19 at 07:51
  • Your answer is very helpful. A lot of machine learning books/tutorials use KNN as the 1st example of supervised machine learning without telling the keys we can get from KNN, So, I am trying to figure it out. – JJJohn Aug 05 '19 at 08:14