9

I am doing some work that requires some estimates of gasoline oil demand elasticity on certain countries. After doing various econometric measures such as instrumental variable, I was able to get reasonable estimate of demand elasticity for most countries, but only for some special ones (mostly in Europe), the values are confusing (positive for example) and the p-val is huge. Is it safe to assume that for these countries the price is just insignificant to Demand? Specifically, I am wondering if I add any other variables, in practice would it only increase the p-val of price and make it less significant?

Say the model I currently have is $$\log Demand = \beta_1 X + \beta_2 \log Price + \epsilon,$$ whereas the new model is $$\log Demand = \gamma_1 X + \gamma_2 \log Price + \gamma_3 Y + u.$$

Further suppose that $Cov(X, log Price) = 0$ (not true), then I think we have the equation that $$\gamma_2 = \beta_2 - \gamma_3\sum_i \frac{Cov(\log Price, Y_i)}{Var(\log Price)}.$$

So if the new variables are correlated to $Price$ then the multicolliearity will just increase their p-val. If they are not correlated, then $\beta_2 = \lambda_2$, and you would theoretically wind up with the same estimates, with a higher SE because of the increase in dgree of freedom, and therefore get a higher P-val. (p.s. I know this argument is not sound in theory, but in practice does it make sense intuitionally?)

Dave
  • 62,186
The One
  • 205

2 Answers2

9

@Dave explains that mathematically p-values can get bigger or smaller with more predictors because adding a predictor changes the model.

But there is a fundamental problem with your argument: it is based on a fallacy about p-values.

Your argument boils down to: "A large p-value means the null hypothesis is true." However, p-values can be large for many reasons, including small sample size: the data are consistent with the null hypothesis as well as other possible explanations. The conclusion we draw from a non-significant p-value is that we don't reject the null hypothesis. "Not reject" is not the same as "accept", in null hypothesis statistical testing at least.

In short, even if you could prove that adding an extra variable increases the p-value of log(Price), you cannot make the interpretation you propose.

So rather than evidence that in some European countries demand is not sensitive to gas prices, it may be the case that you don't have much information about demand elasticity in these countries. [You offer no details about your analysis but a hierarchical model — with partial pooling of price elasticities among European countries — may be an interesting approach to consider.]

You can learn more about significance in a multiple regression in these CV discussions:

Does adding more variables into a multivariable regression change coefficients of existing variables?
What is the effect of having correlated predictors in a multiple regression model?
How can adding a 2nd IV make the 1st IV significant?

dipetkov
  • 9,805
  • This is useful and correct information that I think is better-suited to be a comment, as it doesn’t how introducing additional features affects the p-values of existing features. – Dave Aug 10 '22 at 01:16
  • 2
    @Dave Thanks for the input. I didn't know someone else is writing an answer and I got advised that the preferred CV policy is to write answers, not comments. I'll keep this is an answer since it's not a duplicate of yours. – dipetkov Aug 10 '22 at 01:18
  • I liked this answer as more illustrative as it also explains the case where the p-value increases by adding more predictors (which @Dave's answer overlooked). P.S. [I may not be in that position, but] I further wanted to suggest integrating Dave's answer and simulation in this answer – not sure if it's compliant with CV policy though. – psyguy Aug 10 '22 at 12:27
9

Definitely not, and there is a sense in which this is why we do regression.

An example is analysis of covariance (ANCOVA), which is the usual ANOVA but with an additional variable that typically is called a covariate.

I’ll give an example with two groups.

set.seed(2022)
N <- 100
g <- rep(c(0, 1), c(N, N)) # Group
x <- rep(seq(0, 1, 1/(N - 1)), 2) # Covariate
y <- x + g + rnorm(N, 0, 3)
L1 <- lm(y ~ g)
L2 <- lm(y ~ g + x)         
summary(L1)
summary(L2)

In the first regression, I get a p-value of about $0.025$ for the group variable, while the second regression gives a p-value of about $0.018$ for the group variable.

The reason that I say this is why we do regression is because this is helping us increase the signal-to-noise ratio, thus allowing patterns (signal) to become apparent. If your threshold for evidence of a difference is a p-value under $0.02$, then the first regression gives insufficient evidence that there is a difference between the groups. However, once we use that covariate to explain some of the variability (noise), the pattern jumps out from the background.

EDIT

While this was not the original question, for completeness, I will add that adding a variable does not have to lower the p-value. One way for this to happen is if the new variable is correlated with an existing variable.

library(MASS)
set.seed(2022)
N <- 100
X <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.9, 0.9, 1), 2, 2))
x1 <- X[, 1]
x2 <- X[, 2]
y <- x1 + x2 + rnorm(N, 0, 3)
L1 <- lm(y ~ x1)
L2 <- lm(y ~ x1 + x2)  
summary(L1)
summary(L2)

This goes from a tiny p-value on the order of $10^{-12}$ to a p-value above our usual threshold of $0.05$.

Dave
  • 62,186