6

I'm using supervised classification algorithms from mlpy to classify things into two groups for a question-answering system. I don't really know how these algorithms work, but they seem to be doing vaguely what I want.

I would like to get some measure of confidence out of the classifiers. I can get "real-valued predictions" from the classifiers. These appear to be values of what I would call a link function. Here's some sample output from my system.

Predictions as to whether an answer is correct
for random data from various models

Results for one run
----------------------------------------------
Model Result Confidence? ("Real value")
----- ------ ---------------------------------
 SVM  [True, 0.10396502611075412]
 FDA  [True, 3.3052963597375227]
SRDA  [False, 0.34205901959526142]
 PDA  [True, 3.8857018468328794]
----------------------------------------------

Results for another run
----------------------------------------------
Model Result Confidence? ("Real value")
----- ------ ---------------------------------
 SVM  [False, -0.0059697528841203369]
 FDA  [False, -0.15660355802446979]
SRDA  [False, 1.2465697042600801]
 PDA  [True, 0.23122963338708608]
----------------------------------------------

The real values are generally more positive for classifications as "True" and more negative for classifications as "False", but the link is a bit more complex than that. Can I turn these real values into confidence measures? If so, how?

And I'm playing with the following classifiers.

  • Support Vector Machines (SVMs)
  • K Nearest Neighbor (KNN)
  • Fisher Discriminant Analysis (FDA)
  • Spectral Regression Discriminant Analysis (SRDA)
  • Penalized Discriminant Analysis (PDA)
  • Diagonal Linear Discriminant Analysis (DLDA)

Update: Having thought about this more, realize that I just need the rank of the confidence actually. Does it seem right the answers with the highest real values be the ones that are most confidently categorized in the True group? I'd still like to understand this a bit better, but an answer to that would be nice in the short term.

Avery Richardson
  • 3,171

1 Answers1

6

Regarding the "Real Values"

The "Real Values" are better called "confidences" or (from my pov the most common term) "scores".

Such scores are often normalized so that they sum up to 1 for all classes. They represent a measure how, well, confident the model is that the presented example belongs to a certain class. They are highly dependent on the general strategy and the properties of the algorithm. For example in KNN the score for a class i is calculated by averaging the distance to those examples which both belong to the k-nearest-neighbors and have class i. Then the score is sum-normalized across all classes.

Regarding your question

I suppose with "converting into confidences" you actually mean "probability estimates". E.g. if an example has probability 0.3 for class "1", then 30% of all examples with similar values should belong to class "1" and 70% should not.

As far as I know, his task is called "calibration". For this purpose some general methods exist (e.g. binning the scores and mapping them to the class-fraction of the corresponding bin) and some classifier-dependent (like e.g. Platt Scaling which has been invented for SVMs). A good point to start is:

Bianca Zadrozny, Charles Elkan: Transforming Classifier Scores into Accurate Multiclass Probability Estimates

EDIT after Question-Edit: @Thomas wrote: Does it seem right the answers with the highest real values be the ones that are most confidently categorized in the True group?

Yes, in general this is correct (with the same argument as above). I suggest to create a ROC - plot to see if this also applies to mlpy - package. I suggest ROCR for this purpose.

steffen
  • 10,367