3

I work in the field of behavioural interventions and I use dynamic models to gain better insight into (explain) the process of behaviour change and to help inform future behavioural interventions (exploratory research using system dynamics as model-based theory building).

I currently choose state space models (SSM) as a suitable class of models I intend to explore. However, given the various flavours of SSM i.e. ARIMA, hidden Markov models, regime switching models, hybrid models; I have a couple of questions.

Should I base my modelling choice on:

  1. a goodness-of-fit (GoF) measure e.g. I obtain AIC and BIC for all models under consideration
  2. or on domain expertise e.g. based on my understanding of the domain, I expect the process to follow a stepwise (HMM) rather than a linear process (ARIMA)

Ideally, I would expect the two model selection strategies to agree. Since the choice of model (and thus, its parameters) will be used in theory building, I was wondering what would I do when they don't.

Additionally, I am aware that AIC can be used as a measure only when comparing models fitted with the same estimation method (MLE) on the same data.

Can I use information criteria such as the AIC to compare an ARIMA(p,d,q) with a HMM? (where d >0 i.e. differenced data) Or do I restrict my model space to ARMA and other flavours of SSMs. Thanks!

(I am aware of What are disadvantages of state-space models and Kalman Filter for time-series modelling?, I do not intend to comment on the features of a particular modelling framework, I am more curious to identify an objective measure that might help guide my modelling choice)

Community
  • 1
discretetimeisnice
  • 457
  • 1
  • 4
  • 12
  • AIC works whenever the dependent variable is exactly the same (which is tricky to ensure e.g. when the dependent variable gets truncated when fitting autoregressive models); you don't need the estimation methods of the alternative models to match. – Richard Hardy Aug 19 '16 at 15:18
  • I was under the impression that model comparisons based on AIC are only valid when the competing models are fitted by matching estimation procedures (procedures which are based on maximizing the likelihood). I got this impression after reading the original paper (which I am yet to completely digest). Do you have a reference which states otherwise? – discretetimeisnice Aug 20 '16 at 13:12
  • In my understanding, it is not about how the model was estimated (minimizing least squares, maximizing likelihood, etc.) but whether likelihood can be calculated for the fitted model. If it can, it can be compared using AIC (which is an estimate of the likelihood for a new sample, accounting for any overfitting of the model in the original sample by applying a penalty). – Richard Hardy Aug 20 '16 at 13:18
  • Thanks, Richard, I chose to go with AIC (and BIC) as informal measures of model selection, while at the same time using domain expertise as a basis. In my case, both strategies agree on a model. – discretetimeisnice Aug 22 '16 at 12:15

0 Answers0