7

I have created a model to predict the number of people with a certain characteristic (Y) based on predictor variables $X_1$, $X_2$, $X_3$, $X_4$. The model is a multiple linear regression and both the predictors and the outcome variable have been log transformed, that is my equation looks like: $ln(Y)=a +b*ln(X1)+c*ln(X2)+...$

The aim of the model is to then be applied to a dataset for which we have $X_1,X_2,X_3,X_4$ but need to predict Y (in it's original form). Therefore, I need to backtransform the outputs for Y from the model. I understand that a simple reversal of $e^{ln{Y}}$ isn't appropriate as this does not take into account the error terms within the model, and so including a correction for this is necessary.

I have tried a correction term of the form $exp{0.5*variance}$ as per Miller's bias correction using the below code but this gives me wildly unlikely outputs and so I have assumed is not correct:

exp((summary(model)$sigma)**2)*exp(data$model.prediction)

where

  • data is the dataset used
  • model.prediction is the outcome for each case based on the model,
  • The model is called model.

I am struggling to find the correct R code to make this correction. Is there a package that supports back transformation and if so does anyone know what this code is and how to compute the input terms within it?

Ferdi
  • 5,179
Hannah
  • 73

1 Answers1

7

You didn't give any details about why you think the outputs are wildly unlikely, but my guess is that your errors are not normally distributed. That "smearing adjustment" (bias correction) you're using is only valid if the errors are normal.

There is a more general smearing adjustment you can use, which is easy to implement. If I recall correctly, and I think I do, the steps are:

  1. Compute $\exp(X\hat{\beta})$, i.e. the retransformed but unadjusted prediction.
  2. Regress $Y$ against $\exp(X\hat{\beta})$ without an intercept. Call the resulting regression coefficient $\gamma$.
  3. Compute the adjusted retransformed prediction as $\gamma \exp(X\hat{\beta})$.

It's about the most intuitive thing you can do--forget the theory based on the normal distribution and just estimate the multiplier that gets the job done.

For details, see Duan, Naihua. “Smearing Estimate: A Nonparametric Retransformation Method.” Journal of the American Statistical Association, vol. 78, no. 383, 1983, pp. 605–610. JSTOR, JSTOR, www.jstor.org/stable/2288126.

The Laconic
  • 1,584
  • Hi The Laconic - thanks so much for this response. I had indeed assumed normality of residuals. If I were to check the normality I assume the simplest way of confirming this would be to plot the residuals and assess the shape of their distribution. Is there any way to assess the normality in R using something more rigorous i.e some test I can make? Assuming it's not normal it looks from the above like I essential want to assess the magnitude of the disparity between the observed Y and the back-transformed prediction and then apply that to the new data? I'll take a read of the reference now! – Hannah Aug 10 '18 at 13:51
  • You can look at a histogram of the residuals along with the density of a normal distribution with the same mean and variance; you can look at a quantile-quantile plot; you can run a Jarque Bera test (among others I am sure). – The Laconic Aug 10 '18 at 16:44
  • Great - thanks so much for confirming! This has been so helpful :) – Hannah Aug 14 '18 at 09:24
  • 2
    "That "smearing adjustment" (bias correction) you're using is only valid if the errors are normal." Both the cited paper by Duan et al. (1983) as well as the paper by Newman MC (1993) disagree. In fact, Newman writes: "If the residuals were not normally distributed, then the 'smearing estimate of bias' would be recommended [...]". – COOLSerdash May 26 '20 at 20:44
  • I just tried this approach and it does not seem to agree with Duan's method implemented in software or done by hand. I believe if you replace step (2) with a regression of exponentiated residuals from the first regression on a column of 1s, everything should go through. – dimitriy Jul 24 '20 at 23:49