7

I do not study statistics but engineering, but this is a statistics question, and I hope you can lead me to what I need to learn to solve this problem.

I have this situation where I calculate probabilities of 1000's of things happening in like 30 days. If in 30 days I see what actually happened, how can I test to see how accurately I predicted? These calculations result in probabilities and in actual values (ft). What is the method for doing this? Thanks, CP

Rob Hyndman
  • 56,782
CptanPanic
  • 213
  • I think it would be ideal if you could present some of the data you finally have. At least for me, that would help a lot in thinking about an answer. – Henrik Sep 01 '10 at 16:13
  • The actual probability after the fact is either 1 or 0. So for example For test 1: I say 75% of a 1, and ends up a 1, test 2: 35% ends up 0, etc. How can I quantify the accuracy of my estimate? – CptanPanic Sep 01 '10 at 18:14

2 Answers2

12

In their classic book on the Federalist papers, Mosteller and Wallace argue for a log penalty function: you penalize yourself $-\log(p)$ when you predict an event with probability $p$ and it occurs; the penalty for it not occurring equals $-\log(1-p)$. Thus, the penalty is high when whatever happens is unexpected according to your prediction.

Their argument in favor of this function rests on a simple natural criterion: "the penalty function should encourage the prediction of the correct probabilities if they are known." Assuming the total penalty is summed over all predictions and there will be three or more of them, M&W claim that the log penalty function is the only one (up to affine transformation) for which the "expected penalty is minimized over all predictions" by the correct probabilities.

Following this, then, a good test for you to use is to track your accumulated log penalties. If, after a long time (or by means of some independent oracle), you obtain accurate estimates of what the probabilities actually were, you can compare your penalty with the minimum possible one. The average of that difference measures your long-run predictive performance (the lower the better). This is an excellent way to compare two or more competing predictors, too.

whuber
  • 322,774
11

What you're looking for are called Scoring Rules, which are ways of evaluating probabilistic forecasts. They were invented in the 1950s by weather forecasters, and there has been a been a bit of work on them in the statistics community, but I don't know of any books on the topic.

One thing you could do would be to bin the forecasts by probability range (e.g.: 0-5%, 5%-10%, etc.) and look at how many predicted events in that range occurred (if there are 40 events in the 0-5% range, and 20 occur, then your might have problems). If the events are independent, then you could compare these numbers to a binomial distribution.

Simon Byrne
  • 3,486
  • 19
  • 31
  • 1
    could you elaborate on how the bin counts would be compared to the probabilities? I presume you have something quantitative in mind. – whuber Sep 01 '10 at 20:35
  • 4
    @whuber: the probabilities can be plotted against the empirical frequencies to get what's called a calibration plot; the points should be roughly along the diagonal. I don't know if one could do a more quantitative analysis, this isn't an area I know a whole lot about, just a few things i've picked up here and there. – Simon Byrne Sep 01 '10 at 21:10