1

I've built a model that tries to evaluate the percentage chance of winning for all NHL games. For example:

  1. Ana vs Bos (Bos- 57% chance of winning)
  2. Chi vs Mtl (Mtl - 51% chance of winning)
  3. LA vs Det (Det- 80% chance of winning)

The model is updated daily to take into account what happened in the latest games. At the end of the season, how can I validate my model against the actual results?

If my percentages were a fixed set (55%, 75%, 95%), it would be easy to validate. My model should be right 55% of time for all games that have a 55% chance of winning. My model should be right 95% of time for all games that have a 95% chance of winning.

But since my percentages are all different I don't know how to deal with them. For example, I have predicted percentages like: 51.5%, 53,2%, 54.9%, 62.4%, 88.1%, etc. These came from my model. I'm using a lot of historical data to predict the winner (past games, goalie stats, active players, home vs away, etc., it is similar to the odd column on this page).

gung - Reinstate Monica
  • 145,122
Max
  • 217
  • 1
    What do you mean that they are "all different"? – Tim Aug 18 '16 at 14:04
  • @Tim ex: 51.5%, 53,2%, 54.9%, 62.4%, 88.1%, etc.. – Max Aug 18 '16 at 14:20
  • 2
    But why that happens? What are those numbers? – Tim Aug 18 '16 at 14:21
  • These are coming from my model. I'm using a lot of historical data to predict the winner (past games, goalie stats, active players, home vs away, etc). Similar to the odd column on this page => https://www.teamrankings.com/mlb-win-picks/ – Max Aug 18 '16 at 14:34
  • It would be better if you could edit the information from your final comment directly into the question itself. – Silverfish Aug 18 '16 at 15:11

2 Answers2

2

The situation here is often covered under the heading of scoring rules. See for example this question and answer. One popular way of doing this is the Brier score. The Brier score is basically the average squared discrepancy between what you predicted (which is a proportion) and what did happen (which is 0 or 1).

Community
  • 1
mdewey
  • 17,806
  • Thank you, ,my predictions are very similar to weather predictions :) Rain = Win, No rain = Loss – Max Aug 19 '16 at 13:42
1

You can use cross entropy. See the wikipedia page especially the part on logistic regression. The idea is if Boston wins you evaluate $\ln(x)$ and when Boston losses you evaluate $\ln(1-x)$ where $x$ is the probability your model outputs. You can combine this into one formula: $y\ln(y) + (1-y)\ln(1-x)$. $y$ is the actual result (win or loss). You can use this to evaluate your model and use this as your loss function.

www3
  • 671