Comparing a model's forecasts to observations

Question

I have done an experiment where I have made forecasts and I want to measure how well I did in comparison to observations.

My first approach was a goodness of fit using Excel and the R squared value, however I have been asked to take the standard deviation, or 68% confidence interval of each to determine how well you reproduce the original values.

Could someone tell me how to do this?

This is just as a small snapshot of the data. I have hundreds of values over several categories.

I could take the Model - Observed and divide by Observed and then average it but then what about the variation? My variance, if that is the right term is higher. I need some method of quantitatively assessing how well my model reproduces actual measurements.

Would be grateful for any advice.

Thank you

Answer 1

$R^2$ is not often used as a measure of accuracy in the forecasting community, although it is equivalent to the Mean Squared Error.

There are many established forecast accuracy measures. What you are proposing ("take the Model - Observed and divide by Observed and then average it") is a Percentage Error, and if you take absolute values, you arrive at the mape.

Note that different accuracy measures will be minimized by different functionals of the unknown future distribution. So you should think about what functional you want to elicit and choose a corresponding accuracy measure. See Kolassa (2019, IJF) and references therein, especially to papers by Gneiting, for details, or this earlier thread for the MAPE.

Regarding

I have been asked to take the standard deviation, or 68% confidence interval of each to determine how well you reproduce the original values

I am at a bit of a loss as to what this is supposed to mean. It may mean calculating accuracies for each category (as above), then summarizing it not only by averaging, but also by looking at quantiles, e.g., what looking at the 90% quantile of MSEs. This may make sense, because a forecasting method may work on average but break down completely in problematic cases. You may want to inquire further what is expected from you.