4

I have built two regression models to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. One is a Gaussian model, the other is a Poisson (both using a log link function) both trained on the same dataset of some 30k observations.

My initial confusion: on the training data, the Poisson regression achieved a much better fit as measured by pseudo-R², but had higher RMSE (and mean absolute error). Having read this I understand why.

My question now - when comparing goodness of fit on the test data, what's the appropriate measure to use? RMSE will necessarily favor OLS on the training data, so it doesn't seem like a fair comparison on the test data either. A logarithmic score is valid for Poisson because I have probabilities for each integer outcome but not applicable to OLS.

Some extra information from comments:

Number of observations: About 35k in the training set, and another 10k for test. What was done? My approach had been to look at RMSE and MAE (and inspect a lot of plots) but I was just a bit wrong-footed when my Poisson model - with its much better fit to the training data - nevertheless ended up with higher RMSE and MAE than the Gaussian model.

kjetil b halvorsen
  • 77,844
Tom Wagstaff
  • 245
  • See https://stats.stackexchange.com/questions/142338/goodness-of-fit-and-which-model-to-choose-linear-regression-or-poisson/142353#142353 – kjetil b halvorsen Oct 05 '18 at 12:53
  • Thanks @kjetilbhalvorsen but I've already adopted a log link function across both models. I'm now on to the secondary bit - which is modelling the disturbance term the right way, and what's getting closer to the mark in terms of predictions... – Tom Wagstaff Oct 05 '18 at 14:12
  • So then, why not just use the test set to compare predictions with reality (well, data) and see what does best? You are not limited to just one loss function, you can 1) plot, 2) estimate bias, 3) estimate MSE, 4) estimate mean absolute error, or whatever other error metric which makes sense in your application. You can also compare loglikelihood on test data (a version of log score which makes sense for both models). – kjetil b halvorsen Oct 05 '18 at 14:46
  • Thanks @kjetilbhalvorsen - these are good suggestions. My approach had been to look at RMSE and MAE (and inspect a lot of plots) but I was just a bit wrong-footed when my Poisson model - with its much better fit to the training data - nevertheless ended up with higher RMSE and MAE than the Gaussian model. – Tom Wagstaff Oct 05 '18 at 15:27
  • Let us continue this discussion in chat. – Tom Wagstaff Oct 05 '18 at 15:36

1 Answers1

1

That should be dictated by the use of the model. If used for prediction in future, choose one (or more) measures that is actually relevant for that use. R-squared are typically not good measures for this use, MSE (RMSE) or MAE (Mean Absolute Error) are typically better, and you can compute both.

As you have a big holdout sample, you can just use that to compare the two models on your chosen metric of prediction quality.

You can not use AIC/BIC or such in this situation, as the likelihoods are defined with respect to different dominating measures (Lebesgue measure, count measure).

kjetil b halvorsen
  • 77,844
  • The Lebesgue measure is for the OLS model that assumes a continuous (normal) response variable, and the count measure is for the Poisson model that assumes a discrete (Poisson) response variable. – Dave Oct 23 '19 at 13:12