Confidence interval does not contains 0, should we consider the p value? How?

Question

Here is the summary of my linear model:

Call:
lm(formula = weight ~ height, data = height and weight)

Residuals:
    Min      1Q  Median      3Q     Max 
-10.267  -4.267  -1.267   6.455  11.538 

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)   
(Intercept)       20.4878     5.8335   3.512  0.00158 **
height             0.3195     0.1447   2.208  0.03596 * 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 6.703 on 27 degrees of freedom
Multiple R-squared:  0.1529,    Adjusted R-squared:  0.1215 
F-statistic: 4.873 on 1 and 27 DF,  p-value: 0.03596

And here is my confidence interval for $\hat{\beta}_1$: 0.02260012 0.61639988 at $5\%$ level.

As you can see $0 \notin $CI at $5\%$ level. Also, the $p$-value for the test $H_0: \beta_1=0$ turns out to be $0.03596 $ which is in between $0.01 <0.03596 < 0.05$.

How do I interpret this?

The p-value < 0.05 should mean that there is moderate evidence against the null hypothesis $H_0: \beta_1=0$, also, $0$ is not in CI. So should I reject the fact that $\beta_1=0$ or should I say the evidence is not strong enough?

Furthermore, what would happen if in the same situation, I would get $0 \in CI$ but still $p-$value <0.05?

What about $p-$value >0.05 but $0\in CI$?

Answer 1

Cases where a) the 95% confidence interval doesn't include zero, and cases where b) p <= 0.05, don't always coincide precisely.

For some discussion on cases with the two-sample t test, see perhaps Cumming and Finch, 2005, Inference by Eye.

But also there are different ways the p value may be determined. For example, to assess effects in a model, Wald tests could be used, or likelihood ratio tests could be used, and their results may not be precisely equal.

Likewise, confidence intervals can be constructed in different ways. For example, determining confidence intervals by bootstrap is relatively common now, and there are actual several different bootstrap approaches that could be employed. Again, the results of the different methods won't necessarily yield precisely identical results.

Usually different statistical approaches will yield similar results, but especially in borderline cases where you are using hard cutoff (e.g. p <= 0.05 and p > 0.05 yield drastically different conclusions), different approaches may yield different results.

So what is one to do? A good practice is to decide ahead of time what statistical approach to use, and how you will interpret the results. If you want to use confidence intervals, use confidence intervals. If you want to use p values, use p values.

But in all cases, remember to look at the size of the effects you are observing (in this case, Beta1 or r-squared), and the practical implications of the results. For this model, Is a Beta1 of 0.3 practically meaningful? Is an r-squared of 0.15 practically meaningful?

Answer 2

First of all, it's a 95% not 5% confidence interval, and for the confidence interval (of the slope - since you use the CI ̂ 1±⋅(̂ 1)) you would say: "I am 95% confident that the true slope is between 0.02260012 and 0.61639988, for the current sample." Performing a hypothesis test would yield rejecting the null, so, there is a difference. However, your r-squared indicates your model only accounts for 15% of the variation. You either need to throw out your model or look for the lurking variable that accounts for the remaining 85%. Additionally, the p-value for height indicates that it is statistically insignificant.