Why don't we always fit fully-interacted models?

Question

As far as I know:

1. y ~ x + z means y is dependent on x and z variables
2. y ~ x : z means y is dependent on the interaction between x and z
3. y ~ x * z means y ~ x + z + (x : z)

It looks like * already covers + as well as :. The question is why don't we use y ~ x * z much but y ~ x + z in functions like lm()? Wouldn't we get more accurate results if we used * all the time?

Answer 1

So let's assume that Y depends on A, B, C and D and we have 30 observations.

The appropriate formula is not Y ~ A * B * C * D because then lm will have to find 16 coefficients:

> attr(terms.formula(y ~ A *B*C*D), "term.labels")
 [1] "A"       "B"       "C"       "D"       "A:B"     "A:C"    
 [7] "B:C"     "A:D"     "B:D"     "C:D"     "A:B:C"   "A:B:D"  
[13] "A:C:D"   "B:C:D"   "A:B:C:D"

When would estimating 16 coefficients from 30 observations be sensible? How do you explain the intuition behind A:B:C:D to anyone?

I assume this question comes from a machine learning perspective where data is often assumed to be plentiful and complex models just need to be guarded from overfitting. Well, making simple models is useful in statistics when data is not plentiful, it's useful when you are guarding against overfitting, it is useful when you have a good idea about the data generating process and you want to investigate that specific problem and in many other places.

And do not get me started with the number of coefficients to be computed once we take dummy coding into consideration. If you asked 200 people who all have a gender (2 dummies or more), all have some education on a 8 level scale (7 dummies), all come from one out of 5 countries (4 dummies), all are more or less religious on a 5 point scale (4 dummies) and all have Big Five personalities (5 predictive values). Are we really interested in finding an interaction term for women with a university degree from Spain who are not religious times all five of their personality values? I think everyone can see how this is quickly going to be ridiculous.

On the other hand, sometimes my income is just the some constant times the hours I work in my first job plus some constant times the hours I work on my side job. There is really no point multiplying my work hours. And if you did, you'd compute something with the unit "Dollars per square hours". You do not want to make that the default.

Answer 2

This is quite a deep question, and it is a matter of much debate (here on CV, too). It depends on your goals:

• If your goal is prediction, it is best to use as much information as available. The only possible problem might be overfitting, which can be checked with cross-validation or separate hold-out data.
• If your goal is explanation, a simpler model will be easier to understand as long as the most influential effects are included.

Thus, for predictive purposes, it is generally even better not to assume a linear relationship at all, but to use spline regression, LOESS, or Random Forests (if you have tons of data).

For a polemical discussion of this topic, see

Leo Breiman: "Statistical modeling: The two cultures." Statistical Science 16.3, pp. 199-231 (2001)

And for a more sober discussion:

Galit Shmueli: "To explain or to predict?." Statistical Science 25.3, pp. 289-310 (2010)