One hot encoding with too many features (~ 10,000)

Question

I am building a model to predict time off and sick leave for a specific employee.

Each of the employees has one row per day from 01/01/2013 to 31/12/2018 in the dataset flagged with 0 or 1 (if that day was taken as a time off or sick day). I am using information like location, age, job position, etc.

I was thinking to use employee id as a feature to train the model. Some users have obvious patterns, e.g, taking Monday off for some consecutive weeks. On the other hand, I feel like I do not have enough information as to have one model per employee, specially if they have being hired recently.

After researching a bit, seems that the only encoding that would make sense for employee id would be 'One Hot Encoding'. However, this would generate up to 10,000 features which doesn't seem very optimal.

Do you have any ideas that can help me build this model in a better way?

Thanks in advance

Answer 1

I think you are on the right track. What you are looking for is what is known as fixed effects in econometrics/statistics. Your have a model in which features X determine outcome y (so y = bX + u). On top of that you have unobserved heterogeneity (each individual employee) in your model. For each employee, the relation y <-> X may be a little different (fixed effect per employee), so that your model becomes y = b1X + b2Z + u, where Z denotes employees.

See here for an overview and here for an R implementation.

You can solve this by adding one dummy/indicator/one-hot per employee (known as least squares dummy variable fixed effects). So technically speaking, each employee gets his/her own intercept in the model. Thus the Z above is a massive matrix of one-hot encoded employee labels.

There are solutions in Stata (areg) and R (felm) which "absorb" the massive dummy variable set, so that it does not bother you when looking at the regression coefficients. If you are not interested in statistical inference (but only in making predictions), you will not look at the coefficients anyway.

I don't know the datails of your data (i.e. observations per employee), but from what you say, adding one dummy (aka one-hot feature) per employee would be the right way to go to control for unobserved heterogeneity of the employees.

Answer 2

You could create embeddings for the employees. If you are using NN, just add an embedding layer for the employees. Or, you could use something similar to a collaborative filter to find embeddings for the employees. e.g, Create a matrix E = Employee x Day of the month, where each row is an employee and each column is a day of the month and

\begin{align} E_{ij} = 1 \quad &\text{if employee i worked at the day j}\\ E_{ij} = -1 \quad &\text{if employee i got a time off or sick leave } \end{align} then for each month, you can use, for instance, ALS (Alternating Least Squares) to create embeddings for the employee. You could average the embeddings out for several months and use them in your main model. This approach could also give you some kind of score for each employee that could be used in your model.

Answer 3

If you have enough relevant data, try clustering employees in unsupervised fashion, in order to replace employee ID by their class number; then, one-hot-encode the classes (or use another encoding technique: frequency encoding maybe?). Controlling the number of classes gives you control on the number of features after one-hot encoding, that is surely convenient. Also, new employees can easily be assigned to a cluster, which allows you to immediately be able to make predictions over them (you don't have to collect new data and re-train the model).

Note: A naive alternative would be to engineer some features. For instance, you mentioned taking Mondays off: just create a feature that gives the presence rate on Mondays. But this approach would likely lead to overfitting.