Does including more predictors make existing predictors less likely to be statistically significant?

Question

While listening to a talk I heard a researcher speak about the following approach to doing a multiple regression in the context of an observational study:

1. Subjectively assess all possible predictors and interactions for how well he thinks they will predict the outcome variable. Based on these considerations, rank the predictors/interactions.

2. Subjectively decide on a cutoff beyond which the predictors/interactions seem sufficiently unimportant that he doesn't want to include them.

Both '1' and '2' are done without data snooping, i.e. without actually running any analyses or looking at the data beyond the labels of the variables.

When asked why he takes this approach he says that if he adds too many predictors/interactions it makes it less likely that the preceding ones (i.e. the ones he feels are most subjectively important) will be statistically significant.

I'm somewhat interested in commentary on whether this is a sensible way of selecting predictors for a regression model, on whether such a strong emphasis on statistical significance is wise, etc.

However, what I'm most interested in is whether this researcher's claim is true. Is it true that all else being equal including one more predictor or one more interaction will make it less likely that the previously-added predictors/interactions will be statistically significant? Why/why not?

You can assume for the sake of argument that whatever predictors are selected there isn't going to be any serious problem with multicollinearity.

Answer 1

Altham in a paper in Biometrika available here showed that, to quote her, "... one reason for preferring a model with fewer parameters rather than one with more parameters is that [...] estimates of quantities of interest may thereby be more precise." She provides a proof of this result which she felt was a well-known result (in 1984) but for which she had been unable to find a proof.

Answer 2

I cannot comment, therefore I am putting my thoughts as an answer. I remember Andrew Ng said in this course that he learnt, over the time, not to trust his own gut when it comes to data analysis/machine learning. I would say that I would definitely trust Professor Andrew in this context. My personal experience (which is not very substantial) showed there are people who have a "gut" feeling about the data, mostly developed by working on same dataset (these data changes over time e.g. number of pageviews for a website) over a long period of time. However, even they are not right in many (and I repeat, MANY) cases. In research, I don't think anyone works for years over same dataset and therefore, going with gut feeling is very unreasonable in my mind. In addition, he is not even doing any scatter plot (disclaimer: I inferred that from your description) and therefore, judging importance of a feature subjectively is defnitely not logical.

As for the last part of the question, I am quite confident that if the predictors are not strongly correlated, adding one additional predictor would NOT signficantly change the statistical significance. It may slightly change the coefficient, but not by a large margin. You can see page 122 of this book by Trevor Hastie, Robert Tibshirani and Jerome Friedman where the authors showed a nonsignficant effect can be caused by correlated features (which mean, in the presence of other features, one feature which is important by itself can be dropped).

Answer 3

The steps the researcher outlines follow what I have been taught, though for the sake of expedient study design rather than concern over differences in significance value. Essentially when conducting a multiple regression, each variable you include controls for that variable which, if significant, would otherwise contribute to the error term. In effect each coefficient measures the ceteris paribus effect of the associated independent variable upon the dependent variable outcome (Wooldridge, 2009).

If it appears to be non-significant, it can be removed through backwards selection, otherwise it presumably should be included in the regression model. Changes in estimated effects due to omitted variables are often the result of confounding bias, which can introduce bias into estimates through an association between one or more remaining independent variables and the error term in the model (Mamdani et al, 2005).

So essentially, the answer appears to be that if your effects are changing, this is likely the result of relationships which are being ignored by the model. In the case of confounding IV analyses can be helpful. One word of caution would be reading too much into statistical significance and p.values. Variable selection is best informed through some intuitive potential causal link or previous research, and omitting variables is a tricky business (Sterne and Davey Smith, 2001).

References

Mamdani M, Sykora K, Li P, et al., 2005, “Reader's guide to critical appraisal of cohort studies. 2. Assessing potential for confounding.” Br Med J 2005; Vol.330, pp.960–962.

Sterne J, Davey Smith G., 2001, “Sifting the evidence - what's wrong with significance tests?” Physical Therapy vol.81, pp.1464-1469 (reprint of article 249) and the American Statistical Association statement and the 21 supporting papers)

Wooldridge, J. M. 2006. Introductory econometrics: a modern approach. Mason, OH, Thomson/South-Western.

Answer 4

Adding more predictor always increase the R-square of the model but not adjusted r-square. If the added variable has multi co-linearity issue with other predictor, you will have problem.