How much data are sufficient to train my machine learning model?

Question

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.

My colleague (who is a machine learning expert) said that, in his opinion, the arguably most important practical aspect of machine learning is how to understand whether you have collected enough data to train your machine learning model.

This statement surprised me, because I had never given that much importance to this aspect...

I then looked for more information on the internet, and I found this post on FastML.com reporting as rule of thumb that you need roughly 10 times as many data instances as there are features.

Two questions:

1 - Is this issue really particularly relevant in machine learning?

2 - Is the 10 times rule working? Are there any other relevant sources for this theme?

@Emre Thanks! Can you also suggest me some papers or any material to read? — DavideChicco.it, Jun 28 '17 at 02:14
This will usually be covered alongside cross-validation and other model validation techniques in your textbook. — Emre, Jun 28 '17 at 04:32
The 10 times rule is great if you can achieve it, but it is just not practical in some business settings. There are many situations where the number of features is much greater than data instances (p>>n). There are machine learning techniques designed specifically to deal with these situations. — acylam, Aug 29 '17 at 15:59
If you need a detailed explanation which can help you to understand the learning curve graph check this out: https://www.scikit-yb.org/en/latest/api/model_selection/learning_curve.html — shrikanth singh, Nov 11 '19 at 17:11

score 8 · Answer 1 · answered Jul 07 '17 at 09:07

The ten times rule seems like a rule of thumb to me, but it is true that the performance of your machine learning algorithm may decrease if you do not feed it with enough training data.

A practical and data-driven way of determining whether you have enough training data is by plotting a learning curve, like the one in the example below:

The learning curve represents the evolution of the training and test errors as you increase the size of your training set.

The training error increases as you increase the size of your dataset, because it becomes harder to fit a model that accounts for the increasing complexity/variability of your training set.
The test error decreases as you increase the size of your dataset, because the model is able to generalise better from a higher amount of information.

As you can see on the rightmost part of the plot, the two lines in the plot tend to reach and asymptote. Therefore, you eventually will reach a point in which increasing the size of your dataset will not have an impact on your trained model.

The distance between the test error and training error asymptotes is a representation of your model's overfitting. But more importantly, this plot is saying whether you need more data. Basically, if you represent test and training error for increasing larger subsets of your training data, and the lines do not seem to be reaching an asymptote, you should keep collecting more data.

In learning_curve function should I pass X_train, y_train: Only train subset or X, y: the entire dataset — Rookie_123, Dec 03 '18 at 15:45
This curve is built as a result of applying cross validation as you increase the number of samples. Therefore, you need the entire dataset. — Pablo Suau, Dec 11 '18 at 18:47

score 6 · Answer 2 · answered Jun 26 '17 at 23:39

Yes, the issue is certainly relevant, since your ability to fit the model will depend on the amount of data you have, but more importantly, it depends on the quality of the predictors.
A 10-times rule might be a rule of thumb (and there are many others), but it really depends on the predictive utility of your features. E.g., the iris dataset is fairly small but easily solved, because the features yield good separation of the targets. Conversely, you could have 10 million examples and fail to fit if the features are weak.

Thanks! Can you also suggest me some papers or any material to read? — DavideChicco.it, Jun 28 '17 at 02:14

How much data are sufficient to train my machine learning model?

2 Answers2

Linked