When computing the Pseudo R-squared value in quantreg in R or statsmodel in Python, what is an acceptable range to justify goodness of fit? Also, what is the functional form of Pseudo R-Squared computed in those packages?
Asked
Active
Viewed 632 times
1
-
One potential answer to the form of r-squared in quantreg: https://stackoverflow.com/questions/19861194/extract-r2-from-quantile-regression-summary – Sal Mangiafico Nov 17 '22 at 16:45
-
Another potential answer concerning r-squared in quantreg: https://stats.stackexchange.com/questions/129200/r-squared-in-quantile-regression – Sal Mangiafico Nov 17 '22 at 16:47
-
You should also be able to use Cox and Snell or Nagelkerke or Efron pseudo r-squares. These should be easily obtained from the model object from quantreg. – Sal Mangiafico Nov 17 '22 at 17:03
-
1As to the question about an acceptable range for r-square or pseudo r-square measures, there really is no such thing as a guideline for an "acceptable" range. Expectations are really dependent on the field of study and specific application. In a controlled chemistry experiment, you might expect an r-square of 0.99. When dealing with biological systems or human behavior, even quite small r-squares may be encouraging. – Sal Mangiafico Nov 17 '22 at 17:09
1 Answers
1
As far as what constitutes a good value for a performance metric, this answer explains why that is context-dependent, so there is no hard rule. This makes a lot of sense. While one might view $R^2$ -style metrics as being like letter grades in school where $R^2=0.9$ is the kind of score that seems like an $\text{A}$-grade that makes us happy while $R^2=0.4$ is the kind of score that seems like an $\text{F}$-grade that makes us sad, I remember celebrating a $41\%$ on an extremely challenging exam in graduate school that qualified as one of the higher scores in the class, so even letter grades in school are not as simple as $90/80/70/\text{etc}$.
As far as what constitutes an $R^2$-style metric for quantile regression, it depends on what you want your metric to tell you. My take on the usual $R^2$ is that it is a comparison of the performance of your model on some metric of interest (sum of squared residuals, for the usual $R^2$) to the performance of a "must-beat" model. With that in mind, the typical loss function for quantile regression, much like how the typical loss function for linear regression is the sum of squared residuals, is the quantile loss ("pinball loss" in many circles), $L_{\tau}(y, \hat y)$ in the language below.
$$ l_{\tau}(y_i, \hat y_i) = \begin{cases} \tau\vert y_i - \hat y_i\vert, & y_i - \hat y_i \ge 0 \\ (1 - \tau)\vert y_i - \hat y_i\vert, & y_i - \hat y_i < 0 \end{cases}\\ L_{\tau}(y, \hat y) = \sum_{i=1}^n l_{\tau}(y_i, \hat y_i) $$
As I discuss here, a reasonable $R^2$-style performance metric for a quantile regression would compare the pinball loss of your model to the pinball loss of a "must-beat" model that always predicts the $\tau^{th}$ quantile of all $y$ values combined together. This is more-or-less implemented in Python using sklearn.metrics.d2_pinball_score, though I have some issues with how sklearn performs and documents the calculation. (I have strong feelings about how to calculate the $R^2$-style metrics when it comes to out-of-sample performance, and the sklearn $D^2$ function suffers from the same issues I bemoan in the link. Further, sklearn remarks that the $D^2$ is the proportion of pinball loss explained by the model. This alludes to how $R^2$ is a measure of the proportion of variance explained by a linear regression model, though this breaks down in all but the nicest of circumstances (1)(2), and I expect the same to happen for $D^2$.)
Dave
- 62,186
-
can you also frame this in terms of log-likelihood measures since quantile regression equates to MLEs from a Laplace distribution? – Frank Harrell Oct 05 '23 at 11:18
-