4

I believe I have rather simple question but I would like to make it right.
I have already asked question, however I am not sure whether I did everything correct or there is a mistake in the answer (probably the first one, but still I cannot find it).
To make it short: I would like to check how well my model predicts outcome of a game between two players - and that is why I am trying to calculate R^2. What I have done is based on this answer.
Firstly, I calculate ESS:

ess = 0
for each game in my database:
   ess += (predictedOutcome - realOutcome)^2

The real outcome is either 1 (if win), 0.5 (if draw) or 0 (if lose). To calculate what was the average real outcome, I do:

 averageRealOutcome = (wins + 1/2*draws)/(wins + draws + loses)

The result is always about 0.5 And then for each game I do a loop

tss = 0;
for each win
   tss += (1 - averageRealOutcome)^2
for each draw
   tss += (0.5 - averageRealOutcome)^2
for each lose
   tss += (averageRealOutcome)^2

Then, once more according to the previous answer, I try to calculate R^2:

R^2 = ESS/TSS

Unfortunately the value that I get is above 1, while I believe it should be between 0 (very bad) to 1 (cool). Adjusting R^2 does not do the trick and i can see no difference at all.

  1. Does it in general make sense what I try to do?
  2. If it does, where is my mistake? Or maybe my interpretation is bad? If not, can you provide and resources where I can read what should I do?
Community
  • 1
mkk
  • 153
  • sorry, I was confused myself about your notation, so I removed my answer. I don't like R2 anyway :-) – Nick Sabbe Sep 07 '11 at 16:31
  • @Nick apologizes if I do not follow standards, thanks for spending your time and writing it though! – mkk Sep 07 '11 at 16:34
  • anyone? should I rephrase the question? – mkk Sep 07 '11 at 22:53
  • We don't create new questions to address ones that are unanswered: please just edit your old one. – whuber Sep 08 '11 at 15:22
  • 1
    Note that this is not an $R^2$ because your "TSS" is not the (usual) total sum of squares, nor is it even a reduced sum of squares. – whuber Sep 08 '11 at 15:25
  • @whuber I am not sure how to edit it. I believe that everything is important. According to what you said, does it mean that I cannot use R2 in my case, or I do it wrong? – mkk Sep 08 '11 at 15:30
  • There will be a set of links below the question that include, at a minimum, "link edit flag". Click on "edit". – whuber Sep 08 '11 at 15:33
  • @whuber I meant i do not know what should be edited form of the question, not that I do not know where I can find this option :) I have already edited it many times to make it as clear as possible and not sure how to make further improvements – mkk Sep 08 '11 at 15:55
  • Sorry about the misunderstanding. The description of what you're doing is opaque. Moreover, the code shown is not executable and, as pseudocode, is incomplete and ambiguous. More precision in describing what you're doing, together with a representative sample of your data, might clarify things and inspire someone to offer some insight. – whuber Sep 08 '11 at 16:07

2 Answers2

5
  • See http://en.wikipedia.org/wiki/Coefficient_of_determination
  • Be careful about notation. Use tot, reg, err.
  • From your first code snippet, ess is SS_err. But in the last code snippet, ESS is SS_reg.
  • The actual definition of R2 is 1 - SS_err / SS_tot.
  • If you use this definition and get R2 < 0, that's not a problem. R2 is only guaranteed to be between 0 and 1 for linear models.
  • Use this definition of R2 or a generalization of R2 (also in the Wikipedia article).
Jessica
  • 1,231
2

Since in your predictive model the design is very like in discrete choice models within the case of binary response variables (logistic regression for example would be relevant here), the problem could be approached in the same manner.

First of all you may consider the generalized $R^2$. Note the last suggestion to scale the $R^2$ by the $R^2_{max}$ that insures the modified coefficient is below $1$.

You may also consult the classical text book in econometrics (I used Greene Econometric Analysis Chapter 21.4.5, but a link from SAS help page seems also relevant) regarding the goodness-of-fit for binary response models:

  • McFadden's likelihood ratio index $LRI = 1 - \frac{\log L(\theta)}{\log L(0)}$ the coefficient is pointed to have no clear interpretation though, but this one is a common build-in solution in most of the statistical software
  • Another interesting suggestion is to look either on the average probability of correct predictions as in Ben-Akiva and Lerman: $R^2_{BL} = 1/n \sum_i (y_i \hat F_i - (1- y_i)\hat F_i)$, where $\hat F_i$ is the estimated probability (the estimated value you use to compare with the threshold $0.5$).
  • In unbalanced samples when the number of wins is much different from the number of losses the $R^2_{BL}$ underestimates the smaller group, so you may introduce a measure that compares average correct and incorrect cases $\lambda = (average \hat F_i | y_i = 1) - (average \hat F_i | y_i = 0)$ Cramer's $\lambda$ heavily penalizes incorrect answers.
  • Note that for the unbalanced datasets it is recommended to be cautious about the naive threshold value of $0.5$ since in that case a smaller group risks to be unpredicted too often.
  • The other suggestions for the $R^2$ augmentation are in the abovementioned SAS help page (to be frank, I have never used one thus has no clear opinion regarding them, but the Cragh-Uhler 1 is exactly the generalized coefficient of determination from the wiki page).
Dmitrij Celov
  • 6,275