3

I have recently been bootstrapping the confidence intervals of a neural network model estimated to data.

I execute the following psudo-code, which seems similar to previous bootstraps I have done:

  1. Take all observations of data $x_i$, estimate $\hat{y} = \hat{\theta}(x_i)$
  2. Resample $N$ observations from the set of $x_i$ with replacement. Estimate $\tilde{y} = \tilde{\theta}(x_i)$. Repeat until we have $M$ different versions of $\tilde{y}_m$
  3. Calculate the standard deviation of everything in $\tilde{y}_M$ as $\sigma_{\tilde{y}}$ and then the confidence interval of $\hat{y}$ is $\hat{y} \pm z * \sigma_{\tilde{y}}$ where z is chosen to be an appropriate value from a standard normal distribution, usually 1.96 for 95% CI.

See here for more reading on bootstrap intervals.

At the same time, I see other approaches that use the quantiles of $\tilde{y}_M$ in order to construct intervals of some sort, such as here and here. They do this instead of the $\hat{y} \pm z * \sigma_{\tilde{y}}$ I have come to expect. The intuition for such an argument is very strong and seems valid - but the approach is unfamiliar. What's going on with the use of quantiles? Is something else being calculated (prediction intervals, etc.) that are similar in flavor but are not the same?

RegressForward
  • 1,368
  • 2
    Davison and Hinkley's book Bootstrap Methods and their Application presents a number of approaches to bootstrapping (not exhaustive but both parametric and nonparametric cases are discussed) and it does discuss bootstrap prediction intervals and nonparametric regression models; this might be helpful as a reference, for all that the various pieces are not all in the same places, since the relevant concepts should carry across. – Glen_b May 12 '22 at 01:50

1 Answers1

7

A possible alternative is to identify and form an efficient model generating a set of iid residuals. Generate a forecast using the model and then use the distribution of residuals as the basis of the montecarlo creating limits that are not presumptively based upon a distributional assumption. Use this empirical distribution to offset the expected value. This method allows the identification and incorporation of possible unusual pulses and the option to incorporate (allow for) predicted pulses into the forecast distribution. This is the suggested approach of Prpf. Sam Savage as an extension to his book "The Flaw of Averages".

IrishStat
  • 29,661