11

How do we rigorously define the term "unpredictable" in cases of point and density prediction?

The term "unpredictable" is employed in various contexts, e.g.

  • "the outcome of a toss of a fair coin is unpredictable",
  • "increments of a random walk are unpredictable" or
  • "in an informationally efficient market, price changes are unpredictable".

These statements are not literally true, as one can always provide a prediction, however inaccurate. I can make a point prediction (the coin will turn up heads; the increment will be 0.18879; the price change will be £0.12) or a density prediction ($P(\text{heads})=0.51$, the increment is N(0,0.4), the price change is ...).

Intuitively, "unpredictable" means something like "cannot be predicted more accurately than by some simple/naive/natural benchmark" such as "the best density prediction for the coin toss is $P(\text{heads})=0.5$". But then we need to define what we mean by "best" when evaluating density predictions.

So, how do we rigorously define the term "unpredictable"?

Richard Hardy
  • 67,272
  • Positive variance? – Dave Jul 01 '23 at 19:32
  • @Dave, in my experience that is not enough; see my comment under Aksakal's answer. – Richard Hardy Jul 01 '23 at 19:36
  • Related (non-duplicate): https://stats.stackexchange.com/questions/549914/are-randomness-and-probability-really-logically-dependent-notions – Galen Jul 01 '23 at 19:39
  • Where would you use a rigorous definition? I think it is correct that anything can be predicted — ie for any event, someone may offer a prediction — but many predictions are unreliable. And I think it obvious that no rigorous definition will match all the ordinary uses of the word very well. – Matt F. Jul 01 '23 at 19:52
  • 1
    @MattF., one example of use: when teaching the efficient market hypothesis in a financial econometrics class, I would like to explain it well. For that, I need to understand rigorously what unpredictability means. This is not the only example, thus my question is quite general. Also, I am not trying to match the ordinary use of the word; rather, I would like a definition matching the term's use in statistics and (not too peculiar) applications of statistics. – Richard Hardy Jul 01 '23 at 19:58
  • 3
    I wouldn’t use the word “unpredictable” in teaching the EMH. The Wikipedia article on it (https://en.m.wikipedia.org/wiki/Efficient-market_hypothesis) only uses the word “unpredictable” to warn against its use. Similarly the CFA curriculum (https://www.cfainstitute.org/en/membership/professional-development/refresher-readings/market-efficiency) distinguishes weak, semi-strong and strong forms of the hypothesis without using the word “unpredictable” at all. – Matt F. Jul 01 '23 at 20:46
  • @MattF., thank you, that may be a way out. Still, trying to avoid the term may be tricky at times. Also, textbooks by eminent scholars such as Campbell, Lo & MacKinlay use this term, so the students are likely to encounter it sooner or later. (That was the financial econometrics bible 27 years ago, and newer textbooks such as Linton's "Financial Econometrics: Models and Methods" build on it.) – Richard Hardy Jul 02 '23 at 09:43
  • 1
    Campbell, Lo & Mackinlay’s first page on EMH credits interest in it to Fama (1970), and points to a more explicit definition by Malkiel (1990) replacing unpredictability by the impossibility of making economic profits. They also trace the weak-semistrong-strong articulations of the EMH, about the possibility of superior returns, to Roberts (1967). So econometrics has moved past “unpredictability” as a key concept for the EMH — if students encounter literature emphasizing it, that emphasis has been dated for most or all of the time that the EMH has been prominent! – Matt F. Jul 02 '23 at 11:05
  • @Richard Hardy: Yes, one always can provide a prediction but I think the notion of unpredictable is whether one's predictions actually do better than a guess. Whether it's a coin flip or a price change or any other event, I would claim that an event is unpredictable if , in the long run, one cannot do better in terms of predictive power ( assume that we have a metric that measures predictive power ) than someone else who is guessing. – mlofton Jul 22 '23 at 06:36
  • 1
    @mlofton, for this definition to work, I think we need to define guessing and the metric of predictive power. – Richard Hardy Jul 22 '23 at 06:42
  • 2
    Revisiting this (interesting) question, I think I have to object to the idea of an unpredictable stock market (alluded to in the third bullet point). I invest in stocks specifically because they tend to gain value. They don’t gain value every day or even every year, but they do have an upward trend. That hardly strikes me as unpredictable. – Dave Jul 22 '23 at 19:18
  • I mean the following: Suppose we were trying to check if the residuals from a random walk model were unpredictable. One way to predict would be to use a model to predict direction so up at $t+1$ or down at $t+1$. The "guesser", on the other hand, would have access to a roulette wheel with equal numbers of reds and blacks. He would spin the roulette wheel and if he got a red, he predicts up and if black, he predicts down. Then, at the end, one would calculate the percent correct for both the modeller an the guesser. One could use a test of two proportions to see if proportions are the same. – mlofton Jul 22 '23 at 22:55
  • If one cannot reject the null that the proportions are the same, then the residuals are unpredictable. Just an idea. But that's how I would think of unpredictability – mlofton Jul 22 '23 at 22:56
  • @mlofton Why should the wheel have equal numbers of blacks and reds? – Dave Jul 22 '23 at 23:01
  • So, that the guesser person has an equal chance of predicting up or down, namely 0.5 each. The argument is that, if the modeller can't do better than a pure guesser ,then his-her model has no predictive power. – mlofton Jul 23 '23 at 00:21
  • Dave: I was thinking about your question more. If one biases the roulette wheel so that it's not half the time red and a half the time black, then the guesser could achieve more than 0.5 accuracy by chance. For example, say he always chooses up and it just happens that a lot of residuals are positive. So, by choosing $p = $ 0.5 for black and red, one would expect the guesser's accuracy to approach 0.5 after enough trials ?. If it did have such behavior, then one could turn the hypothesis test into a one proportions test where one tests the modeller for $H_o: p = 0.5$ versus $H_1: p > 0.5$. – mlofton Jul 23 '23 at 05:05
  • Dave: I may have above wrong. There must be literature by smarter people who have thought about these things way more than I have. My thinking is that one should have a null hypothesis and an alternative but even that's tricky. Suppose one uses unpredictable as the null and predictable as the alternative, I'm not sure now whether one uses a 2 proportion test or uses a test of one proportion. – mlofton Jul 23 '23 at 07:52
  • But above is kind of tangential to Richard's original question Somehow, given the data, one needs to create a "random" set of predictions and a set of predictions that are from an actual model, Next step is create a null hypothesis and alternative hypothesis. Then test given the two sets of predictions by constructing the test-statistic. So, my only point is that "unpredictable", atleast from the modelling viewpoint, and under some accuracy metric, can be quantified and tested for. The tricky part is the "random" predictions. The Diebold-Mariano paper probably discusses this. – mlofton Jul 23 '23 at 08:00
  • Anyway, to close the deal on my earlier roulette wheel comment, by using the same number of reds and blacks, I think I was trying to create truly random predictions. – mlofton Jul 23 '23 at 08:02
  • @mlofton So is uniformity the key to unpredictability? That seems consistent with entropy from information theory, though I do wonder what happens when the space of possible outcomes is unbounded (e.g., no theoretical cap on how much an investment can can gain). That still doesn’t work for me on the context of unpredictable investments, though, as markets tend to gain over the long term. I think this is part of what makes this question so interesting! – Dave Jul 23 '23 at 08:08
  • 1
    yeah. you would have to account for the risk premium for investing in stocks. But that's what those "factor models" are supposed to do. So, maybe predictability could be benchmarked against some factor model ?. I was thinking more along the lines of say an intraday strategy where one is doing many trades and number of wins/number of trades is CRUCIAL. One can probably also use a proportions test for that case ? But that's true that, in an investment management type scenario, the premium needs to be accounted for which is another tricky problem. – mlofton Jul 23 '23 at 08:32
  • Last thing to Richard mainly: I think Granger must discuss predictability or lack thereof in his granger causality material. I never looked at it much but I know that he's using prediction to decide whether variable X granger causes Y. So, the notion must be atleast mentioned. – mlofton Jul 23 '23 at 08:39
  • 2
    My 2 cents: I'm glad I don't have to deal with this question... I don't use the term "unpredictable", and I don't know whether it's a good term to have in our vocabulary. I would prefer something like "can't be consistently predicted better than using ". This immediately gives the benchmark to compare to and draws attention to the fact that we can indeed always give some kind of prediction. – Stephan Kolassa Jul 25 '23 at 13:18
  • @StephanKolassa, thanks for the 2 cents! I like your suggested approach. Would it help to refer to a loss function in addition, or does "unpredictability" under one loss function imply "unpredictability" under all other [sensible] ones? – Richard Hardy Jul 25 '23 at 13:20
  • That is actually an interesting question. (a) Yes, the loss function is important in the sense that it governs what the "simple benchmark" would be. (In my example, I pretty much took the loss function for granted as a piece of background.) ... – Stephan Kolassa Jul 25 '23 at 13:47
  • 1
    ... (b) No, "unpredictability" in "my" sense under one loss function does not imply "unpredictability" under another one. Assume outcomes are a mixture of 60% zeros and 40% something strictly positive that depends on predictors. Under the MAE loss, the optimal prediction is always zero, the in-sample median, so under MAE, the outcome is "unpredictable". But under MSE loss, we may be able to improve on the climatological prediction, i.e., the in-sample mean, by quite a bit. – Stephan Kolassa Jul 25 '23 at 13:49
  • @StephanKolassa, thanks! I think your comments provide a good starting point for a rigorous answer. – Richard Hardy Jul 25 '23 at 14:07
  • 4
    Trying to rigorously define a term that is up to that point used in an informal manner will normally change the term, and will not cover all of its uses. (Of course there can be reasons to do it anyway, but I think it's good to have in mind what can and what cannot be achieved in this way.) – Christian Hennig Jul 25 '23 at 15:00
  • 1
    If "unpredictability" is interpreted as positive variance, I wonder if the distinction must be made between zero-variance variables and strict constants. Loss functions interpretation does not make such a distinction either. But naive intuition wants to make a distinction between a coin with $P(H)=1$ and a coin with heads on both sides. [P.S. I guess, I am just confusing something being predictable and something being deterministic. But a similar conflation somewhat takes place in Aksakal's answer and the comment under it] – paperskilltrees Jul 25 '23 at 15:14
  • 1
    @paperskilltrees That’s provocative, and I mean that as a compliment! However, that would seem to give special meaning to the H that “would” have been T. – Dave Jul 25 '23 at 15:16
  • I think saying "unpredictable" in contexts where we actually mean "not able to be accurately predicted consistently" is an unfortunate lapse in common usage that will be hard to correct. Perhaps always explain to students that the term is just being used as a type of shorthand. – Graham Bornholt Jul 26 '23 at 01:07
  • Related: https://galenseilis.github.io/posts/train-predict-assume-depend/ – Galen Jul 26 '23 at 15:41

2 Answers2

7

I see several levels to unpredictability.

  1. Positive variance: If the outcome is uncertain, there is a degree of unpredictability. If the outcome is certain, then there is no unpredictability.
  2. Positive variance, but zero conditional variance: Maybe an outcome is uncertain, but it becomes certain once you observe some other information (e.g., regression features).
  3. Can’t do better than predicting based on a uniform distribution on the space of outcomes: If the distribution gives equal likelihood to every outcome, then you truly have no idea about which will happen. (I’m not quite sure how to think about this when the space of outcomes is unbounded.)
  4. Can’t do better than always making the same naïve prediction, such as always predicting the majority category: Then there is a limit to how good the predictions can be, and it takes minimal skill to reach that limit (basically no skills required to buy an S&P 500 index fund).

I think the last one seems consistent with efficient market hypotheses. We might think it is impossible to do better, consistently, than buying and holding the entire S&P 500, but we do not claim that buying and holding the S&P 500 is foolish or has zero expected gain. Indeed, doing so seems to be a wealth pump!

Bringing rigor to the last two requires a few definitions to be made.

DO BETTER is defined by some kind of utility function that will depend on what is valued. In the context of finance, perhaps this is the returns on an investment.

NAÏVE PREDICTION requires a careful definition of a benchmark or baseline model, as is mentioned in a nice comment by Stephan Kolassa.

My 2 cents: I'm glad I don't have to deal with this question... I don't use the term "unpredictable", and I don't know whether it's a good term to have in our vocabulary. I would prefer something like "can't be consistently predicted better than using insert simple benchmark prediction". This immediately gives the benchmark to compare to and draws attention to the fact that we can indeed always give some kind of prediction.

Dave
  • 62,186
  • 1
    Number 2 nicely represents the notion of pseudrandom numbers. They can be unpredictable even with the generative algorithm known, but conditioning on the seed number you get exactly the right answer every time time; zero conditional variance. (+1) – Galen Oct 21 '23 at 16:19
2

I think you're making this more complicated that it needs to be. "the outcome of a coin toss is unpredictable" - mean exactly what it says, that the outcome is random and is not determined fully by the state of the world before the coin toss. If you're interested in the probability, i.e. population parameter, then it is predictable: it's whatever you measure it because it doesn't change.

When we say the stock price is unpredictable we mean just that: we can't beat the random toss. It doesn't mean we can't calculate the distribution parameters with some precision.

Aksakal
  • 61,310
  • 2
    I have encountered situations where the vague intuitive interpretation does not cut it, so I am after a rigorous definition this time. Also, I doubt "the outcome is random and is not determined fully" is a good characterization of unpredictability. If there was a model that predicted the stock price better than a random walk model, the stock price would be called predictable; this is what you find in the finance literature. Similarly, tomorrow's temperature is predictable, as we have models that do better than the long-term average temperature or some other naive benchmark. – Richard Hardy Jul 01 '23 at 19:24