1

I have 1200 data points and I'm building 6 different time series models. One with 1200 data, another model with 1100, 1000,..., 700 and then I would choose the best model but I don't know how to select between different data lengths. I'm doing so because my data have time based trend and I don't know how many data should I use to make the models.

I've tried to use $$AIC=-2\log(L)+2k$$

where $k$ = # of Parameters, $L$ = Likelihood, $N$ = Data Length

To avoid AIC always selecting the model with minimal data length, I've used the logarithm in the base N, but it still prefers to choose the model with lower N.

kjetil b halvorsen
  • 77,844
Bernardo Braga
  • 125
  • Possible duplicate: http://stats.stackexchange.com/questions/251056/comparing-differences-of-aic-of-different-data-sets. As per the comments, there are subtleties involved in this endeavor. It would be much easier for you if you could simply train the different models on the same data, rather than datasets of different size. Is this not an option? – jwimberley Jan 09 '17 at 02:02

1 Answers1

3

You can't use AIC to compare models that were fit to different data. Comparing AICs amounts to determining which model results is less information loss relative to the true model. When you have different data sets, in addition to the scale issues you point out, there may be different "true" models, so the comparison doesn't make sense regardless of whatever you're doing to fidget with the base of the logarithm.

If you really must compare these models, I suggest some absolute measure of fit, such as an $R^2$ type measure (i.e. squared correlation between the fitted values and the true values) or mean squared error. If the data are binary you could use area under the ROC curve, which has a nice interpretation (the probability that a randomly selected "positive" will have a larger predicted value than a randomly selected "negative")

gammer
  • 1,487
  • yes the prediction is binary, I'll try to measure the area under the ROC (subject that I did not know). Higger is better? – Bernardo Braga Jan 09 '17 at 12:50
  • I found this BriorScore wich I guess is more adequate of what I'm doing: https://en.wikipedia.org/wiki/Brier_score – Bernardo Braga Jan 09 '17 at 13:46
  • That basically looks like mean squared error. Seems reasonable. – gammer Jan 09 '17 at 14:21
  • But there will be no problems with this huge difference between data lengths? the model should 'unfaithfully' adapts better with lower data. I know I'm dividing by N but still... – Bernardo Braga Jan 09 '17 at 15:18
  • No, I think it's fine. The MSE estimates produced from the larger data sets will be more accurate of course, but they are still comparable, unlike the AICs. You can try to get uncertainty estimates for the error rates and incorporate that into your comparison if you want. – not_bonferroni Jan 09 '17 at 18:05