How to find weight by maximizing the rank ordering performance

Question

How to find the weight such that the final value (weighted sum) n range 0-1 represented as probability, as weight1 * var1 + $weight2 * var 2.... that would maximize the rank ordering performance? (AUC or any other performance measure, a new reference).

The logistic regression does not provide the desired solution.

The data I'm dealing with look like this: (with var1...varX is a variable converted to probability measure) so I would like to aggregate the individual measurement of risk across different tools to single risk measure including some confidence interval. The exit variable is target binary response variable that indicate the risk materializing.

How to approach it or which tool to use to achieve this? I'm using r statistical software so suggesting a r package would be also helpful.

Answer 1

You have a straightforward optimization problem. The Optimization CRAN Task View is a very helpful resource.

First off, your objective function takes (in your example) three weight parameters. These are matrix-vector multiplied with your variables. The result is a vector of class membership probabilities. We compare these to the actuals (exit) and calculate the AUC. This can be done using auc() in the pROC package. This entire function needs to be fed into the optimization algorithm.

Next, your optimization is nonlinear (since the AUC does not depend linearly on the weights). You also have constraints: your weights must be nonnegative and less than one. These are box or bound constraints in the Task View. However, we also have the constraint that they must sum to one. This is not a box/bound constraint any more, but a linear constraint, so any tool on the Task View that only discusses box/bound constraints is not helpful.

I settled on the Rsolnp package, not because I have any experience with it, but because it was the first one to look helpful. It looks like it performs minimization (though it apparently nowhere says so explicitly).

Read your dataset (next time, best to provide the data as an MWE, ideally as the output of dput()):

dataset <- data.frame(
    var_prob1 = c(.95,.12,.34,.61,.17,.26,.78),
    var_prob2 = c(.28,.18,.33,.77,.70,.48,.05),
    var_prob3 = c(.77,.74,.47,.67,.14,.38,.43),
    exit = c(0,0,1,0,1,0,0))

Define the objective function - since we want to maximize the AUC and Rsolnp minimizes, we minimize negative AUC:

library(pROC)
objective <- function(parameters) {
    -as.numeric(auc(response=dataset$exit,
        predictor=(as.matrix(dataset)[,1:3]%*%parameters)[,1]))
}

Finally, call the solver:

library(Rsolnp)
n_parameters <- ncol(dataset)-1
result <- gosolnp(
    fun=objective,
    eqfun=function(parameters)sum(parameters),
    eqB=1,
    LB=rep(0,n_parameters),
    UB=rep(1,n_parameters))