Model evaluation metrics for comparing predicted probability accuracy across different datasets?

Question

I'm working on an online model scoring framework, my goal is to be able to understand if my model's predictive performance is degrading week-over-week. I have a classification model (trained on binary target data) which produces probabilities, and I am most interested in assessing the accuracy of those probabilities. For model selection and tuning, i'm using log loss, but I understand that log loss can't be used to compare performance across different datasets (different weeks of predictions). Is there a best practice metric that I can use to effectively compare the accuracy of predicted probabilities, across different datasets? More explicitly, what are appropriate metrics to monitor predicted probability quality over time?

Answer 1

I agree with your statement that comparing log losses $L(p_i) = - \log (p_i)$ across different datasets is not very meaningful if these datasets exhibit different statistical behaviour. See also this post on 'good' log loss values for more background.

Looking at differences

I would suggest to modify the metric you are already using, so that it's interpretation becomes (more) meaningful across datasets. The easiest way would be to switch to loss differences instead of pure losses. If $\bar L$ is the average loss of your model and $\bar L_{base}$ is the average loss of a simple baseline forecast (e.g. frequency of success in last $n$ weeks), then $\bar L_{base} - \bar L$ should be positive most of the time. If you start to see negative values for several weeks, this suggests that your model is no longer useful. Keep in mind that you'd have to check whether the observed sign changes are significant or just bad luck for your model.

Forecast Skill

To refine this idea, you could also scale the average loss difference by the average loss of the baseline model (if it is positive, which it should be for the log loss), i.e. via $$ \bar L_{skill} := \frac{\bar L_{base} - \bar L}{\bar L_{base}} = 1 - \frac{ \bar L}{\bar L_{base}} $$ Then $\bar L_{skill}$ is bounded above by 1 (if your model were perfect) and values below 0 again indicate that your model is no longer useful. Additionally, it quantifies how big the loss difference is compared to the baseline loss and thus gives you a useful interpretation across datasets.

For proper scoring rules (of which the log loss is an example) this quantity is also called skill score, see for instance this paper. In Section 2.3 the authors also state that

If scores [meaning losses in this discussion] for distinct sets of situations are compared, then considerable care must be exercised to separate the confounding effects of intrinsic predictability and predictive performance

which summarizes the reasons why it is probably ill advised to compare losses across different datasets.

Connection to coefficients of determination such as $R^2$

As noted by Dave in the comments, forecast skill relate to McFadden's pseudo $R^2$ and might agree with it, depending on the baseline model. Additionally, if $L$ is the squared error loss and the baseline model is a pure intercept model (i.e. just the mean of the observations), then the in-sample computation of the skill of a model agrees with the usual $R^2$ coefficient of determination from ordinary least squares regression. The same connection arises for the case where $L$ is the absolute loss and the coefficient of determination $R^1$ which is sometimes used in quantile regression.