probably a very basic question -- I am modelling companies' decisions on which mode of payment they use in M&A deals with a help of logit model. I am so far interested only in what variables are statistically significant, I am not trying to create a model for predictions so far.
Is there any use for train, validation and test data sets in statistical inference or are they of use only for machine learning (predictions)?
Thank you, Adam
When doing inference, it is best to use all the data you have, as this will give you more precise estimates in your model.
That being said, there are instances where it may make sense to split your data. If you are interested in generating hypotheses and then testing those hypotheses, splitting your data into a set for investigation and then for inference makes sense. The best example of this is likely when people do something like a stepwise regression. What people should do is split their data, do the stepwise regression to generate hypotheses on what variables are impactful, and then do inference on the other half with the variables the stepwise procedure selected. What you typically see is that people do hypothesis generation and inference on the same data which is a huge no-no.
Prediction counts as a kind of statistical inference, and it's not restricted to machine learning. But when you're looking at mere associations rather than predictions, and you're interested in the statistical significance of these associations, there's indeed no reason to split the dataset into training, testing, and validation sets. (What would you even be testing or validating?)
This is a complex subject. So as to provide a simple answer, I would suggest that you split your data if your data set is sufficiently large (i.e. larger than 5,000 observations).
Kodiologist, you are indeed testing something. You are testing whether correlations are statistically significant or whether they can be attributed to chance. If you run dozens and dozens of models, you will find correlations, but you drastically increase the risk of a false positive.
A simple way to guard against this is to split your data, analyze the training data to find a few correlations that might be statistically significant, and then test this small number of correlations on the test data set.
For more on this very complex topic see: https://web.stanford.edu/~fafchamp/samplesplit.pdf
