0

I need to do a naive time series forecasting as a benchmark. Therefore, I should split my dataset into train, validation, and test sets. If I divide the the dataset into train and test sets, the code will be as follows:

library(fpp2)
library(forecast)
data(ausbeer)
train <- window(ausbeer, start = c(1956,1), end = c(2007,4))
test <- window(ausbeer, start = c(2008,1))
naiveS <- snaive(train, h=1)
accuracy(naiveS, test)[,1:5]

How about if I want to use the data during 2005-2007 as my validation set? Could you please let me know how to compute the model's performance on the validation set? I read this post about sliding/rolling window and expanding window and this page. As I know the tscv is based on expanding window.

e <- tsCV(ausbeer, snaive, h=1)
sqrt(mean(e^2, na.rm=TRUE))

When we use a naive forecasting method, how to obtain the performance on the train, validation, and test sets?

ebrahimi
  • 291
  • It depends on what you mean by "validation" (and therefore also, "test") set - different people use the terms in different ways. If by "validation set" you mean "data for the final evaluation", then just adjust your windows and use what you call test. If you mean "intermediate evaluation set for tuning hyperparameters", then there is no such set, because the naive forecast does not have any hyperparameters to be tuned. – Stephan Kolassa Feb 04 '23 at 06:17
  • @StephanKolassa Thanks. I mean "intermediate evaluation set for tuning hyperparameters and selecting models". I need a naive forecast as benchmark to compare this naive model and other different models to each other, so I should obtain the performance of naive model on the validation set. – ebrahimi Feb 04 '23 at 13:35
  • If the validation set is an intermediate set that you only use to tune hyperparameters, then I don't see why you need the performance of the naive forecast on this set, because the naive forecast does not have any hyperparameters to tune. And on the other hand, if you do want the performance of the naive forecast on this set, you can simply set the window accordingly. I think I'm not completely understanding your question. – Stephan Kolassa Feb 04 '23 at 20:06
  • @StephanKolassa thanks. I want the performance of the naive forecast on this set, but I don't know how to set the window accordingly? – ebrahimi Feb 04 '23 at 21:21

0 Answers0