2

Let's say you have a set of potential explanatory variables (e.g. p = 8) that you think are important to explain your response variable ($Y$) but your sample is too small to include them all in the same model (e.g. n = 50) but you do it anyway, would this overfitted model be worth something to generate new testable hypotheses?
For instance, if the model's summary shows that $X_{2}$ and $X_{5}$ have a large (positive or negative) effect on $Y$, would it be appropriate to conclude something like: "the results suggest that these two variables may be influencial, so we recommend testing the effects of $X_{2}$ and $X_{5}$ on $Y$ in a controlled experiment"?

I know I could use a penalization method instead but I still would like answers to these questions.

Fanfoué
  • 631
  • 5
  • 16
  • What is the purpose of making this claim in the first place? Your proposal is to do a follow-up experiment where you measure all 8 variables anyway. Why not use the first experiment + other available knowledge to estimate the sample size needed for a second experiment? – dipetkov Jun 18 '22 at 18:41
  • Well, in this example, the idea was not necessarily to measure all 8 variables but to control for other influencial variables through the experimental design of the follow-up study and solely focus on the effects of X2 and X5, but I realise my phrasing was confusing (I'll edit my question). Altogether, I just want to know if patterns emerging from an underpowered model can be used to suggest new hypotheses regarding variables importance or whether overfitting makes the model results untrustworthy even for exploratory purposes? – Fanfoué Jun 20 '22 at 08:19

1 Answers1

1

Essentially, you suggest to (1) do variable selection from an under-powered study and (2) validate the selected variables with a follow-up experiment (presumably, with sufficient power).

Step (1) has the usual pitfalls of variable selection from a small dataset: the most significant coefficients are most likely to be over-estimates of the true effect sizes. This is related to the concepts of type S (sign) and type M (magnitude) errors [1].

Fortunately, if that's the case, step (2) will produce unbiased estimates of the effect sizes of "influential" variables because regression to the mean is a powerful phenomenon.

So you will do good science by externally validating the variable selection process. And in this sense your proposal for small-data driven hypothesis generation is "appropriate", which is not the same as "efficient".

[1] A. Gelman and J. Carlin. Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9(6):641–651, 2014.

dipetkov
  • 9,805