1

I've recently started using LMM which possibly gives me better insight into my DV. Only I got some contradictory data regarding whether some variable is significant or not.

The Us and Hed variables are continuous variables, and the App and Category are both multicategorical (ordinal) data (set as factors in R)

My lmer function:

    xxlmer <- lmer(Us ~ App + Hed + (1|Category), data = dataset)

Now I've noticed that lmer doesn't show the p-value. And I've read this is done for some good reasons. However, I would like to calculate these for use in my thesis.

First I found the following code which translated the t-values to p-values (For which I'm very grateful).

coefs <- data.frame(coef(summary(xxlmer )))
# use normal distribution to approximate p-value
coefs$p.z <- 2 * (1 - pnorm(abs(coefs$t.value)))
coefs

This gives succesfully gives me p-values, which look like this:

            Estimate   Std..Error   t.value        p.z
(Intercept) 2.9044048 0.49348777 5.8854646 3.969374e-09
App1        0.1600932 0.21344810 0.7500335 4.532345e-01
App3        0.3825582 0.20096127 1.9036414 5.695690e-02
Hed         0.3417938 0.09047678 3.7776961 1.582858e-04

However, when I subsequently use the 'xxlmer' in stargazer it also gives me p-values. These are way more conservative, making most insignificant (not even showing one star which is the equivalent of <0.1). I know there is a debate going on about calculating and using p-values, and it's deliberately been removed from the lmer function. But I've always assumed the difference wasn't so big.

Therefore, my question is: Which of the output can I trust and should I therefore use?

amoeba
  • 104,745
Mischa
  • 31
  • 4
    I don't know what xxlmer is, but you can get p-values from lmer if you install lmerTest package. For simple designs these p-values are likely to be rather reliable. Your manual computation uses z-test instead of a t-test and this is unjustified unless you have a huge sample size. Don't do that. – amoeba Oct 23 '17 at 20:50
  • 1
    This is really difficult to answer until you give us some more details about what "'xxlmer' in stargazer" is. I tried googling it but did not immediately see a link that talked about it. – Jake Westfall Oct 23 '17 at 20:56
  • Not an answer, but you can use rstan::stan_lmer with the same formula, use some conservative priors, and you get real samples from the posterior distributions for each of the coefficients, you can translate those to credible intervals, 89% highest density intervals, whatever you wish, and be a Bayesian! – Gijs Oct 23 '17 at 20:57
  • In addition to @amoeba's good suggestion you can also use confint( modelName, method='boot') and directly get bootstrap confidence intervals. – usεr11852 Oct 23 '17 at 23:36
  • Sorry if I was a bit unclear but with xxlmer in stargazer. I meant: I put xxlmer (which I defined above) in the stargazer function. This function translates the R info into latex tables. In doing so it apperently calculates the p-values in some way. – Mischa Oct 24 '17 at 05:49
  • @Gijs what do you mean by rstan::stan_lmer? I I've tried googling it but that didnt help. – Mischa Oct 24 '17 at 05:58
  • @amoeba I tried your suggestion but I believe it's been removed from the lme4 package as I don't succeed in getting a pvalue. Also see https://stats.stackexchange.com/questions/22988/how-to-obtain-the-p-value-check-significance-of-an-effect-in-a-lme4-mixed-mode – Mischa Oct 24 '17 at 05:59
  • 1
    Read help("pvalues"). Not being a purist and generally mainly producing p-values to satisfy coauthors and reviewers, I'm usually happy with using the lmerTest package, which masks the lmer function and its summary and anova methods with a version that provides p-values. – Roland Oct 24 '17 at 06:05
  • 1
    So, just do library(lmerTest) before fitting the model. – Roland Oct 24 '17 at 06:10
  • Mischa, I know that lmer itself does not yield any p-values. As I said, you need to install lmerTest package, load it, and call lmer from that package. It adds p-values to the output. See the third answer in the thread you linked to. +1 to @Roland's comments. – amoeba Oct 24 '17 at 06:41
  • @Mischa, rstan::stan_lmer refers to the function stan_lmer that's part of the rstanarm package. Package can be found here: https://cran.r-project.org/web/packages/rstanarm/index.html. – Gijs Oct 24 '17 at 08:32
  • Thank you all for explaining! It's been a bit confusing but it's working now.

    @Gijs I'm not sure yet what the advantage is of the method you explained but I've put it on my list to go and look up :),

     – Mischa Oct 24 '17 at 19:49
  • So did you try lmerTest? If so, did the p-values agree with whatever you got from stargazer? I don't have any experience with of knowledge of stargazer, that is why I am curious. – amoeba Oct 25 '17 at 08:31
  • 1
    Oh sorry, yes I used the lmerTest which worked fine but it apparently didn't load due to some errors I got. Turns out Stargazer provided me with the correct anwers :) – Mischa Oct 27 '17 at 07:55
  • How do you interpret p.z ? –  Oct 28 '17 at 01:17

0 Answers0