What can be a robust model validation strategy for time series modelling?

Question

I am working on time series data where I am using FB Prophet algorithm. I am willing to build a dashboard to produce validation results to my clients. What could be a possible model validation strategy, keeping in mind that I want to present the status of the model in the form of a Red-Amber-Green (RAG) status.

Answer 1

There are various possibilities, and you will need to tailor your dashboard to your specific application.

You could monitor forecast accuracy, ideally with a holdout set. (In-sample accuracy is a notoriously bad indicator of true forecast accuracy.) For instance, if you have 24 months of history, you could fit a model $M$ to the first 23 months, forecast one month ahead, and compare accuracy on the last month of history. Depending on accuracy, you could display alerts. Then refit $M$ on the full history to give you the "real" forecasts.

How achieved accuracy translates into alerts will need to be governed by your context. There is no external "benchmark accuracy" that is useful as a comparison. You could use an internal benchmark, e.g., derived from a simple method: if $M$ does not improve on a seasonal naive forecast, this may be cause for concern, and if it forecasts significantly worse (what this means is again up to your context) than the naive method, you may want to set up a big red flag. Alternatively, you may be able to figure out when a forecast is "good enough" or "problematical" in terms of the subsequent business decisions made - if the forecasts from model $M$ would lead to the same business decisions as a perfect forecast, e.g., because of logistical constraints or production batching, then the accuracy of $M$'s forecasts is completely irrelevant. You may be interested in Ways to increase forecast accuracy.

Alternatively, you could monitor data quality, or data dynamics, through a kind of process monitoring chart. Use your favorite model ($M$ or a naive model), fit it to the initial 23 months, calculate prediction intervals for the holdout month, and raise an alert if the holdout data from the 24th month lies outside that prediction interval. Set the coverage probability of the predition interval to a level that balances the number of alerts and their usefulness.

You could also monitor the forecasts themselves: if your forecast is suddenly twice as high as all in-sample fits or all historical observations, this may be worth investigating. It may be because of some new and never-before observed combination of predictors, which you may want to take a look at before automatically acting on that forecast.