4

I have a object detection problem which has extremely imbalanced dataset. Lets say there is only one class to detect, say banana or not banana. This detection network will be used in a real case where positive/negative samples ratio is extremely huge, 1:1Million.

What are the best approaches to this imbalanced dataset problem ?

My first idea to train the model with using a dataset of 100:100k positive/negative samples ratio but the model still might be skewed to negative samples which might results in not detecting bananas apparently.

I am using Yolov4 with AlexeyAB's Darknet as a object detection model. Also, background of images are mostly the same.

Lynn
  • 1,707
Uce
  • 41
  • 3
    Maybe better to do anomaly detection or one-class classification. – user2974951 Jul 26 '22 at 12:48
  • 4
    The first step is to determine the costs of false positives and false negative errors. If they are equal it is quite likely that the optimal decision is to classify everything as belonging to the majority class (see https://stats.stackexchange.com/questions/539638/how-do-you-know-that-your-classifier-is-suffering-from-class-imbalance ). Incorporating the misclassification costs into your classifier system is probably all you need to worry about. – Dikran Marsupial Jul 26 '22 at 12:52

1 Answers1

5

Separate the probabilistic modeling step from the decision step.

First, model and predict the conditional probability for a given instance to belong to the target class. This will likely always be a tiny number (and, per the thread Dikran links to, for highly imbalanced data the imprecision of your model may be the dealbreaking aspect here), but it will be slightly higher for certain instances than for others.

Then decide on your decision or action based on the probabilistic class membership predictions. Especially for highly imbalanced situations, we typically have differential actions, or more than one possible decision. If we are looking for a murderer (the murderer incidence is very low) and have someone whose DNA matches that found at the scene, we do not immediately fire up the electrical chair, or even schedule the trial. Instead, we collect more evidence: does the suspect have a motive, or an alibi? If the predicted probability is essentially zero, we look for other suspects; if it is somewhat higher (maybe there was not much DNA to be found), we collect more evidence; if the probability is overwhelming, we arrest the suspect.

At this point, the costs of wrong decisions are relevant. (I don't like the term "misclassification", because as above, there are often more than two possible actions.)

More information here.

Stephan Kolassa
  • 123,354
  • +1 but I'm not convinced that the probability of seeing a banana should always be very low. It would be unusual for me to see a monkey, so the prior probability is low (probably not a million-to-one, but low), but if I see a monkey, I'll probably know it's a monkey. If a banana really stands out from non-banana, a similar argument should apply if the non-bananas look dramatically different from the bananas (so apples and watermelons, not plantains). – Dave Jul 26 '22 at 15:11
  • 2
    @Dave: I think if the bananas stand out very much (i.e., we have a high signal, low noise ratio), the question is much easier, and we are all a lot happier. "Superfragilisticexpialidocious" probably is a word with extremely low incidence, but OCR-and-spell-checking will still easily be able to decypher it, because there is just so much signal. An easier situation all around. – Stephan Kolassa Jul 26 '22 at 17:18