1

Let's say I have two variables and I want to measure the association between them. I've been reading up on running Pearson's correlation vs. Spearman's. I understand that Pearson's measures linear relationships and Spearman's measures monotonic relationships. Here are several thing I've concluded: 1) In cases in which an association seems linear based on a scatter plot, then use Pearson’s (assuming all other Pearson’s assumptions are met). 2) In cases in which an association seems monotonic based on a scatter plot, then use Spearman's (assuming all other Spearman’s assumptions are met). 3) If an association seems neither linear nor monotonic you can basically use either Pearson’s or Spearman’s and the output of Pearson’s will indicate that no linear relationship is present and the output of Spearman’s will indicate that no monotonic relationship is present… which also implies that if an association seems neither linear nor monotonic it would probably be just as fine to not run Pearson’s or Spearman’s at all. Does that all sound correct?

And a second closely related question: What if I look at the scatter plot and it’s not clear whether the relationship is linear or monotonic. This could mean that it seems equally likely that it’s linear or monotonic, or that it seems a bit more likely that it’s linear, or that it seems a bit more likely that it’s monotonic. What is one to do in such cases? Should one just always run Spearman’s in such cases to be safe (given that all linear relationships are monotonic)?

FastBallooningHead
  • 481
  • 1
    #3 is not necessarily true at all: Models assuming linearity can "find it" even when non-linear relationships are at play, same holds true for models assuming montonicity when non-monotonic functions 'truly' describe the relationship. – Alexis Feb 19 '23 at 21:05
  • 1
    If you are interested approaches to understanding how two variables relate which make more general assumptions, check out nonparametric regression models a la Buja, A., Hastie, T., & Tibshirani, R. (1989). Linear Smoothers and Additive Models. The Annals of Statistics, 17(2), 453–510. – Alexis Feb 19 '23 at 21:07
  • 1
    Finally, you might also check out information-based alternatives. For an interestingly different approach than Pearson and Spearman, see Reshef, D. N., et al. (2011). Detecting Novel Associations in Large Data Sets. Science, 334(6062), 1518–1524. – Alexis Feb 19 '23 at 21:10
  • 2
    @Alexis I agree with most of your points, but the measure posted by Reshef et al. was put down rather firmly by Simon and TIbshirani: see https://arxiv.org/pdf/1401.7645.pdf – Nick Cox Feb 19 '23 at 21:47
  • 2
    @NickCox Ah! Good to know. I know that their work was an early effort in a new direction, and had almost certainly been superseded by methodological refinements, but the thing I like about the (pretty short) paper is that it shows a very different approach to conceptualizing and estimating method for answering questions like "How much does knowing about $X$ tell us about $Y$?" Off to get myself edumacated now, thank you! :) – Alexis Feb 19 '23 at 22:07
  • 1
    This is now your eighth question about Pearson &/or Spearman correlations, six in the last 23 hours as of this comment. Since these correlations are straightforward - "Pearson measures the strength of a linear relationship, Spearman that of a monotonic relationship, and correlation does not equal causation" comes close to summing it up, it seems to me this is indicative of some fundamental misunderstanding(s) about them. Perhaps you should save yourself some Q&A effort and find a source that explains them in one block of text... instead of the many, many small ones you are getting now. – jbowman Feb 19 '23 at 22:11
  • 1
    @NickCox Ok, and having now read S & T's very brief critique, I stand by my recommendation to 'check out', but with the added caveat that I am not endorsing the Mutual Information Criterion in practice. :) That said, I will also direct FastBallooningHead's attention to Székely, G. J. & Rizzo, M. L. (2009), Brownian distance covariance, The Annals of Applied Statistics. pp. 1236–1265. – Alexis Feb 19 '23 at 22:12
  • @ Alexis; yes, for # 2 I meant Spearman's. That was a typo. Thanks for pointing that out. As for what you say about # 3: This was more of just making an inference... But also I'm not totally sure how what you're saying differs from what I'm saying. I'm probably misunderstanding. Will try thinking about it some more. And thanks a lot for the links! I will check them out. – FastBallooningHead Feb 19 '23 at 22:38
  • Thanks dipetkov!! I replied there. – FastBallooningHead Feb 19 '23 at 22:42
  • 1
    @Alexis There is now book on distance correlation (plus plus). https://www.routledge.com/The-Energy-of-Data-and-Distance-Correlation/Szekely-Rizzo/p/book/9781482242744 – Nick Cox Feb 20 '23 at 15:59

1 Answers1

2

A linear correlation and a monotonic correlation are quite different things.

You should not be using the data to choose what hypothesis you want to test on the same data. The significance level (and hence p-value) no longer has the correct properties in general. In particular, if you calculate p-values without accounting for the effect of choosing your hypothesis based on the data, the calculated p-values (and the nominal significance level) will tend to be lower than they should be.

For example, imagine you were dealing with a population with either a weak or null linear relationship and smallish samples. Some samples will look more strongly linear, some will look less clearly linear, and some of those will instead suggest a curvilinear, but monotonic relationship. If you pull away a subset of the samples that are more suggestive of a curved monotonic relationship than a linear one, the most "linear-looking" samples (the ones that you're going to reject H0 on) will be a larger fraction of the samples that are left. That is, you will reject a larger fraction of the Pearson tests both when H0 is true and when it's false. This selection process is effectively raising power by pushing up the significance level, but in a way that will typically be hidden when it comes to publication. A similar effect can be described for the fraction of samples you pull out to treat differently, as well.

See here:

https://en.wikipedia.org/wiki/Testing_hypotheses_suggested_by_the_data

This is in effect a form of p-hacking; one might call it "hypothesis shopping", or a version of HARKing. Its impact may be small in some situations and large in others; I don't know quite how large it might be in this particular situation, since the outcomes of the tests on the hypotheses you're choosing among will be positively dependent to some extent, but how much is not going to be clear unless we specify a number of things. In any case, if you're taking a broadly falsificationist approach to a research question (as seems to be the case), it's clearly not proper scientific practice to be choosing which statement you're falsifying after seeing the data you want to use to falsify it.

You should instead be choosing your specific research hypothesis before you see data (indeed, hopefully before even collecting it; you should know what question you're trying to research at the planning stage, unless it's an exploratory study, in which case, you shouldn't normally be dealing in p-values).

If a monotonic correlation was actually something you were interested in, why are you considering a linear correlation? (or, indeed, vice versa).

The usual approaches to generating scientific research hypotheses would presumably all apply here.

[Indeed, how were you doing power calculations and hence obtaining a useful sample size if you don't even know your actual hypothesis?]

If you're not in a position to know which hypothesis you want to test, it seems like you're not really in a position to be doing more than exploratory analysis.

This sort of issue appears to be distressingly common; indeed I have over time gained the strong impression that some areas of research teach students to use very vague hypotheses as a matter of practice. Whether or not that is the unstated policy, I see questions like this in various places on an almost daily basis. This sort of post hoc (after the data are observed) hypothesis selection may well be a substantive contributor to the reproducibility crisis that affects a number of areas of research.

Glen_b
  • 282,281