0

I have incidents VS normal operation of my working environment. It is a skew dataset. My prediction accuracy is 95%.

Question:
1. Is it common practice among data scientist to accept this prediction?
2. Do I have to rework by resample and balance the skewness and train & test again?

I am sorry to ask for partial opinion based question.

Incident VS Normal

ii2
  • 111
  • 5

3 Answers3

1

The first question is very broad, 95% accuracy doesn't say a lot, there are tasks where 99% is standard and there are tasks where an accuracy of 60% is amazing. It is hard to answer this question without more information about your model, dataset and the affects of its outcome. Basically you should ask yourself if it is good enough for your task.

Regarding the second question, I would suggest to use a weighted loss function, where mistakes on the rarer classes have a higher price (it encourages the model to classify them correctly). For optimal performance, you should retrain your model from scratch.

Note for future reference, it is better to present the results in a confusion matrix, than just the final classification (gives more data, especially when we talk about a skewed dataset).

Mark.F
  • 2,220
  • 2
  • 16
  • 25
0
  1. There are a lot of imbalanced datasets such as cancer and anomaly detection datasets. I can say that it is very normal that we work with imbalanced datasets.
  2. If skewness of dataset is high, you can try to increase the number of samples with few data. Or you may consider the weighted loss function to improve the performance of the prediction model.
Nga Dao
  • 185
  • 5
0

Yes. I agree this is a subjective question. Based on the histogram, you have 16500 samples of class 8 and 500 of other classes. The class representation is around 97% and 3%. This is severe imbalance. So without any model, it is safe to always blindly predict the output as class 8. This gives me around 96% - 97% accuracy based on random sampling.

Now getting an accuracy of 95% using ML model doesnt give additional confidence. What is happening is the model got trained looking at class 8 samples most of the time and it will blindly predict class 8 as outcome.

You need to definitely follow undersampling of class 8 or oversampling of other classes (SMOTE).

solver149
  • 111
  • 2