Bootstrap validation for panel data machine learning

Question

A traditional machine learning validation strategy is to train on some data and check performance on some holdout data. When data are time-dependent, an obvious way to proceed is to train on early periods and test on later periods, say train on data from $2000$ to $2011$ and then check performance in $2012$, $2013$ and $2014$ (those are the real years for which I am likely to have data).

A technique from biostatistics is to train on a bootstrap sample of the data and then test on the original data. An advantage of this is that valuable training data are not allocated to a holdout set.

However, when it comes time to bootstrap this, the way to proceed is not clear. Do we bootstrap everything and use the future to predict the past? That seems problematic.

What would be the way to do this for the panel-data I seem to have?

This seems related but not a duplicate.

Answer 1

I disagree that selecting later data as a hold-out set is an obvious way to proceed (with validation) when you have panel data. This is in fact a different and more stringent way to perform model validation. In some literature, these are distinguished as external validation or temporal validation. To illustrate this point, consider that a cubic curve locally approximates a sigmoidal curve prior to the inflection point, but once you sample data over the entire extent of the curve, the quality of fit deteriorates.

If we focus on cross-sectional validation, it's straightforward to conceive of the bootstrap as yet another resampling based approach to developing independent datasets that are expected to have the same internal structure. That is, if you inspect first and second moments comparing datasets that are randomly split p/(1-p) and bootstrap resamples, all univariate and bivariate statistics have the same expectations. The advantage of the bootstrap, which Frank Harrell points out in his text Regression Modeling Strategies, is that many properties of estimators are affected by small samples (feature selection using LASSO, small sample bias in logistic regression, etc.), and so bootstrapping can mitigate these problems by having the same $n$ in the training and validation sets.

It is a sound question, therefore, to ask whether a temporal bootstrap can be conducted for a panel analysis that produces a similar sized $n$ in training and validation sets, which enforces the stringency of a temporal validation, but which averts the issues of reducing the effective sample size. One obvious choice is just to apply the date cutoff and then resample observations to achieve a maximal $n$, another obvious choice is to not do that because the point of the exercise is to impose a needful penalty on model complexity. The correct choice would depend on your specific analysis, and perhaps some simulations to explore the operating characteristics of either approach. Of course, the specifics of bootstrapping panel data is already addressed as per your post.