0

I have this idea in mind, but I am not sure how to implement it. Suppose I have an imbalanced data that I want to down-sample instances of majority class, such that it becomes equal in size to the minority class.

I want to use SVM to select from majority class, it's class support vector(s) and examples near distance such support vectors until majority class examples equal the minority examples.

For example: enter image description here

In the above figure, we have 8 instances of class 1 and 5 instances of class 0 before undersampling. After under-sample, we have 5 instances of class 1 and 5 instances of class 0 so the classes are balanced.

Can someone offer guide how this can be done?

arilwan
  • 273

1 Answers1

2

Firstly, don't undersample unless you have vast amounts of data and need to undersample for computational reasons.

Rather than resampling, use different values of the C hyper-parameter for each class instead. This is equivalent to resampling, but without throwing away any data (just down-weighting it). For details see my paper (with Mrs Marsupial)

G. C. Cawley and N. L. C. Talbot, Manipulation of prior probabilities in support vector classification, In Proceedings of the IEEE/INNS International Joint Conference on Neural Networks (IJCNN-2001), pp. 2433-2438, Washington, D.C., U.S.A., July 15-19 2001.(www,pdf)

There were several other papers written on this topic at about the same time, but that was the easiest for me to find. Most modern implementations of the SVM will support this.

However, the reason for changing the C parameters or resampling is not because of the imbalance, but because false-positive and false negative errors have different costs.

The approach you intend to take is unlikely to work because as soon as you delete some support vectors, that will change the solution simply because the support vectors are necessary to define the optimal hyperplane. This means you will be invalidating much of the theory on which the SVM is based. So I would advise against it.

I don't think SVMs have difficulties with imbalanced learning tasks, and if it is assigning everything to the majority class, it is quite likely that that is the optimal solution for equal false-positive/false-negative costs (see my related question here).

Dikran Marsupial
  • 54,432
  • 9
  • 139
  • 204
  • Many thanks for the detailed answer, I will study the paper cited about. However, I'm not deleting the support vectors, instead selecting the support vectors (and example near the support vectors for majority class) into the training set. In my case, I consider each class to have equal misclassification cost (no such as as concept of interest, only that for data collection reason, one class is under-represented). – arilwan Apr 04 '22 at 18:08
  • @arilwan From the question "Suppose I have an imbalanced data that I want to down-sample instances of majority class, such that it becomes equal in size to the minority class." resampling the data that way is equivalent to changing the misclassification costs. It is not clear why you are downsampling if the misclassification costs are equal. – Dikran Marsupial Apr 04 '22 at 19:22
  • May be my words are misleading, but what I mean is that there is not such thing as concept as concept of interest here. Detection of each class is important, only data for data collection reason, one class is under-represented. – arilwan Apr 05 '22 at 08:42
  • Sorry @arilwan that does not really explain why you want to downsample. – Dikran Marsupial Apr 05 '22 at 08:46