Is it valid to calculate the AUPRC of a model by undersampling one class and then re-oversampling its results?

Question

Given a model (returning scores between 0.0 and 1.0), and

• 1000 positives input examples
• 100000 negative input examples

I'd like to calculate the AURPC.

Calling the model is costly, so I don't want to invoke it on all 101000 (1000+100000) samples. (~2k samples would be ok.)

The "normal" approach would be to evenly subsample from both sets to get 20+2000 samples and just use them.

However, my idea is to do the following (uneven subsampling) instead:

• Keep all 1000 positive examples, but subsample the negative ones to 1000.
• Get the scores from the model for these 1000+1000 samples.
• Repeat the array of the 1000 scores from the negative examples 100 times to blow up the scores again back to 1000+100000 (to have back the original positives-negatives ratio).
• Use these "faked" 1000+100000 scores to calculate the AUPRC.

My experiment (simulation) shows, that the AURPC is more accurate (closer to the one of the full dataset) when using the "1000+1000" (uneven subsampling (with re-upsampling)) approach compared to using 20+2000 (even subsampling).

So my question is: Is my uneven-subsampling-with-re-upsampling approach a generally valid method for cost reduction? (Or was I just "lucky" in my simulations because of the chosen distribution or so?)

Answer 1

Is it valid to calculate the AUPRC of a model by undersampling one class and then re-oversampling its results?

Yes, but only if you're careful in the implementation. Throwing re-sampled scores into scikit-learn's average_precision_score gives overestimates at ~25% relative error for your data. (Overestimation is demonstrated in the first version of this answer, which is based on this older version of the simulation code. I haven't completely ruled out that scikit-learn's AUPRC estimate on the whole sample is biased. Either way, you need to be careful in the implementation.) These issues are well-known enough that there's a paper which proposed throwing out AUPRC in favor of a different metric:

Flach, Peter, and Meelis Kull. "Precision-recall-gain curves: PR analysis done right." Advances in neural information processing systems 28 (2015).

Back to your question. Empirically, the following re-sampling-based estimators are somewhat unbiased:

• $\text{repeat}$: down-sample the more prevalent class, compute predictions, then repeat them back to the original size

• $\text{up-sample}$: down-sample the more prevalent class, compute predictions, then randomly up-sample them back to the original size (from @usεr11852's comment)

• $\text{re-weigh}$: down-sample the more prevalent class, compute predictions, then re-weigh them in the computation of AUPRC (derived in the next section).

My implementations of the pure down-sampling estimator—

• $\text{down-sample}$: down-sample the dataset uniformly at random, paying no attention to the class

—appear to be slightly biased, but I don't know why.

Derivation of $\text{re-weigh}$ estimator

Repeating data can be codified as re-weighing in the AUPRC scoring function. So only the scoring function needs to be modified, not the data. Perhaps this fact saves you some compute for super large-scale evaluations.

Let $P$ represent the universe where data are not re-sampled. Let $Q$ represent the universe where we re-sample. In your re-sampling scheme, $P(Y = 1) = 1\text{k} / (1\text{k} + 100\text{k})$ and $Q(Y = 1) = 1/2$.

Given a threshold $t \in (0, 1)$, call $p_P(t)$ the model's precision on $P$ at $t$, call $r_P(t)$ the recall, and $f_P(t)$ the false positive rate, i.e.,

$$ \begin{equation*} p_P(t) = P(Y = 1 \: | \: \hat{Y} > t) \\ r_P(t) = P(\hat{Y} > t \: | \: Y = 1) \\ f_P(t) = P(\hat{Y} > t \: | \: Y = 0). \end{equation*} $$

The AUPRC score we'd like to estimate is that on $P$ :

$$ \begin{align*} \text{AUPRC} &= \int_{t} p_P(t) dr_P(t) \\ &\approx \sum_{i=2}^{n} p_P(t_i) (r_P(t_i) - r_P(t_{i-1})). \\ \end{align*} $$

In the sum above, $\{t_i\}_{i=1}^{n}$ is a grid of decreasing thresholds over $(0, 1)$, which implies that $r_P(t_i)$ monotonically increases.

The game is to estimate each $p_P(t_i)$ and $r_P(t_i)$ using metrics from $Q$—the universe where it's less costly to compute model metrics. The trick is to realize that $Q$ and $P$ are equivalent when they're conditioned on the class ($Y = 0$ or $Y = 1$), because the class frequency is the only difference between $Q$ and $P$. This fact implies $r_P(t) = r_Q(t)$ and $f_P(t) = f_Q(t)$. The precision calculation requires some manipulation:

$$ \begin{align*} p_P(t) &= \frac{r_P(t) P(Y=1)}{P(\hat{Y} > t)} && \text{Bayes' rule} \\ &= \frac{r_Q(t) P(Y=1)}{P(\hat{Y} > t)} && \text{class-conditional} \end{align*} $$

where

$$ \begin{align*} P(\hat{Y} > t) &= r_P(t) P(Y=1) + f_P(t) (1 - P(Y=1)) && \text{total probability} \\ &= r_Q(t) P(Y=1) + f_Q(t) (1 - P(Y=1)) && \text{class-conditional}. \end{align*} $$

Plugging it back in:

$$ \begin{align*} p_P(t) = \frac{r_Q(t) P(Y=1)}{r_Q(t) P(Y=1) + f_Q(t) (1 - P(Y=1))} \end{align*} $$

This proves that $P(Y=1), r_Q(t)$, and $f_Q(t)$ are sufficient to calculate $\text{AUPRC}$.

Simulations

Code for this simulation is available here.

Let's empirically evaluate the AUPRC estimators on the Beta-distributed predictions from your simulation:

Repeating the estimator definitions (for easier reference):

• $\text{down-sample}$: down-sample the dataset randomly

• $\text{repeat}$: down-sample the more prevalent class, compute predictions, then repeat them back to the original size

• $\text{up-sample}$: down-sample the more prevalent class, compute predictions, then randomly up-sample them back to the original size (from @usεr11852's comment)

  • as suggested in the linked comment, the implementation of the ECDF inverse includes linear interpolation. See the sample method in the code

• $\text{re-weigh}$: down-sample the more prevalent class, compute predictions, then re-weigh them in the computation of AUPRC.

The simulation computes the signed difference between the estimator's observed AUPRC and the "true" AUPRC—that on the full dataset. Each estimator uses the same number of model() calls, i.e., the model compute is held constant.

Here are summary statistics:

       error down-sample  error repeat  error up-sample  error re-weigh
count            500.000       500.000          500.000         500.000
mean              -0.021         0.009           -0.002           0.009
std                0.041         0.027            0.020           0.027
min               -0.094        -0.042           -0.047          -0.042
25%               -0.050        -0.010           -0.015          -0.010
50%               -0.031         0.004           -0.005           0.004
75%                0.001         0.024            0.009           0.024
max                0.220         0.113            0.066           0.113

Notes:

• $\text{up-sample}$ has the lowest bias and variance. It gets bonus points b/c you can repeatedly up-sample, which allows you to be transparent about the variance from up-sampling. Though there's no cheap way to convey the important source of variation—that from the initial down-sampling.

• $\text{down-sample}$ has significantly higher variance than the rest because it only looks at a handful of the less prevalent class. It's also weird b/c the direction of bias depends on the implementation. The first version of my simulation demonstrates overestimation, as does your simulation. Both used scikit-learn's implementation.