9

Suppose you're trying to predict anomalies. That is, consider the case where you have a data set that has a column called result. Suppose the data set has 365 rows and result has a value of 1 in only 12 of those rows and 0 in the other rows.

Now suppose you have another column in the data set called val1. Further suppose that the p-value of the correlation between result and val1 is small (say < 0.05). Note that I'm measuring this using the R cor.test method.

Does this imply that we should be able to predict somewhat accurately the value of result given the value of val1?

I naively assumed it did and used logistic regression to predict but got very bad F1 results. (Basically the logistic regression model always predicted 0 for result and thus there were no true positives.)

Tim
  • 138,066
Paul Reiners
  • 827

3 Answers3

14

In regression, the p-value of a coeficient is the result of performing a hypothesis test about correlation, with the null hypothesis being that the correlation equals zero. Having a statistically significant correlation just means that we have a small p-value; and a very small p-value means that we can be very sure that the correlation is not zero. However, please notice that being sure the correlation is different from zero doesn't tell us anything about how large the correlation is - and it can be very small.

A very small p-value with a small correlation just tells us that we can be sure that our independent variable explains a small part of the variance of our response, and therefore it has very little predictive power.

In summary, it's possible to get a correlation that is both statistically significant and very small. In addition to possible, it's quite common when we have large samples.

Edit to make an addition: This is just an occurrence of the quite common phenomenon of getting a result with large statistical significance but tiny practical significance, that often happens when sample size is large.

For example, when doing a t-test to assess if a drug reduces probability of cancer, we might get a p-value of 0.00001 for a reduction larger than zero at the same time we estimate a reduction of probability of 0.000000001%. We could be very sure that there is a reduction of probability of cancer (based on our p-value) but for any practical purpose that reduction is so tiny that we can see the drug as having no effect.

With correlation it's the same: small p-value and small correlation makes us sure that correlation exists but it's small. However, sometimes correlation is big enough to have practical meaning (independent variable explains a sizeable part of the variance of the dependent variable) but not big enough to have predictive power.

Pere
  • 6,583
4

No. Correlation measures linear relationship between two variables, so if the relationship is not linear it becomes useless. You can easily produce examples where variables are strongly correlated ($r = 0.58; p < 0.001$) while the fit of the regression line to such data is far from "accurate".

Correlation with non-linear data

Community
  • 1
Tim
  • 138,066
  • 8
    I disagree that the linear fit is "useless" if the relation is not truly linear. In your example, using the linear fit is typically better than doing nothing, although clearly suboptimal (although probably worse than doing nothing for heavy extrapolation). If a linear fit on a non-linear relation was useless, pretty much all of statistics would be useless. – Cliff AB Aug 20 '16 at 19:23
  • 5
    @CliffAB you're right I'm exaggerating, but the point is that correlation measures only the linear relation. – Tim Aug 21 '16 at 10:36
2

Late on the post, but to help future generations I will respond. In short, the answer is no. Statistical significance does not tell the researcher anything about a model's predictive ability.

The definition of p-values makes this clear. A p-value represents how likely it is that you the researcher observed a given data set under the assumption that the null hypothesis is true. So even if you reject the null, you really cannot say anything about which model is better at predicting, only that what you observed is more or less likely given the null. A super common mistake is that people assume rejecting the null implies that the alternative is a better model, which is simply not the case.

Perhaps a more complete answer would be this: p-values are not a valid metric for model comparison. To compare models effectively, you need methods like AIC or BIC. One could also ditch frequentism all together and use Bayesian estimation techniques to construct Bayes factors, which would allow you to compare models.

One last (and to some controversial) comment on p-values. Consider this, given enough data, you will obtain significance. This is due to the (silly) nature of the null-hypothesis, which states that an effect is exactly zero. In a vast majority of cases, this is almost certainly not true. There will be some non-zero effect, just a small one. So what p-value thresholds do in practice is set up straw-man requirement that can always be met with enough data. Thus, given that you can always achieve significance, it should be unsurprising that p-values do not really tell you anything about model performance. As far as I can tell, they really do not tell you much of anything. As scientists, we care about estimation and prediction and p-values do not help us do either...

Ben
  • 31