I recently stumbled over a generalisation of $F_1$ score to cases where the model predicts probabilities: $$ F_1 = 2 \frac{\sum y_i \hat{p}_i}{\sum y_i^2 + \sum \hat{p}_i^2} $$ where $y_i \in \{ 0, 1 \}$ are the true class memberships and $\hat{p}_i \in [0, 1]$ the predicted class probabilities.
$F_1$ score is usually expressed in terms of either precision and recall or of true and false positives and negatives. However, with just elementary algebra it can be re-expressed as: $$ F_1 = 2 \frac{\sum y_i \hat{y}_i}{\sum y_i + \sum \hat{y}_i} $$ with $\hat{y}_i \in \{ 0, 1 \}$ being the predicted class memberships.
I actually have two questions:
Simply substituting $\hat{y}_i$ for $\hat{p}_i$ seems straightforward, but is it legitimate?
Where do the squares in the first formula come from? Is it just because, for binary vectors, $\sum \hat{y}_i = \sum \hat{y}_i^2$, or is there a deeper theoretical justification?
I observed that, when using squares in the denominator, the generalised $F_1$ score reaches maximum when $\hat{p}$ is not too far from the true $p$ (in a way, approximating a proper scoring rule):
while for non-squared values the maximum is reached when $\hat{p}$ is either zero or one:
(similar to Brier score vs. absolute loss). Using such scoring has no advantage over the original, binary $F_1$ score. So, are squares in the denominator just a hack to make generalised $F_1$ more useful in practice?
