0

Beginner ML question here. I have a dataframe with a categorical column, a lot of the values are slightly different but essentially mean the same thing. Here's an example of such values:

Wireless Connection
Wireless Connection  Wi-Fi
Wireless Connection Issue 3g
Wireless Connection Issue Wi-Fi
Wireless/Connectivity

What would be the best way to condense/encode these values? Intuitively my thinking is to just find all instances of the values and convert them to single unique value like, Wireless Connection. For what it's worth, I'll be feeding the encoding of these values into some classifier, like a Decision Tree or Logistic Regression to predict if a customer's question in a call to customer service was answered.

kjetil b halvorsen
  • 77,844
Keith Johnson
  • 143
  • 5
  • Since your data is categorical, you don't have implicit relationships defined that would allow you to know that the meaning of values is similar. Would you be able to confirm that similar meaning could possibly be indicated by some sort of string similarity? – mapto May 14 '19 at 13:30
  • 1
    Yes I was thinking of doing something like that; the total number of unique values is around 1,000 so my plan was just to go through them manually and make a judgment on them myself. – Keith Johnson May 14 '19 at 14:07
  • 1
    Maybe human-validating them is a good idea. If new categories might emerge in future, you might consider semi-automating the process using libraries like TextDistance, jellyfish or something of the kind. Also consider different possible distance metrics, e.g. from this historical site. – mapto May 14 '19 at 15:50
  • Another kind of idea is https://stats.stackexchange.com/questions/146907/principled-way-of-collapsing-categorical-variables-with-many-levels which collapses levels not from semantics, but if they give sufficiently similar predictions. – kjetil b halvorsen Jun 26 '19 at 23:50

0 Answers0