1

I have 2 datasets, I am creating models on the 1. one (tree,forest,lasso,stepwise OLS, all cross-validated) and then I have to predict the 'Age' variable I am training the models on for the 2. dataset (not part of the 2. set). What metrics could I use to choose a model that is the best? Do I only look at the MSE, AIC, BIC, R square on the first dataset or should I look at the performance of the prediction as well? If that's possible at all?

JohnFire
  • 133
  • 1
    Some of these can be obtained from test data, and you should probably use these when making decisions, and not pay much attention to these metrics on the train data. – user2974951 Feb 14 '23 at 11:50

1 Answers1

2

As user2974951 writes, do not look at in-sample performance, which is a notoriously poor indicator of true predictive performance.

There are many different point forecast accuracy measures; here is a list and discussion. Be aware that your models will likely try to give you unbiased expectation forecasts, which minimize the MSE (and variants thereof). If you want to minimize the MAPE, you should tell your models about this, because you want different forecasts then!

Stephan Kolassa
  • 123,354
  • I know that in-sample is not the best way to decide, but since I don't have the actual Age values for the dataset where I use the predictions on, I can't really measure the model on that dataset – JohnFire Feb 14 '23 at 13:04
  • In that case, I would recommend you use a holdout set from your dataset 1, or cross-validation. And then hope that your model is equally applicable to dataset 2. – Stephan Kolassa Feb 14 '23 at 13:06