Fine-tune random forest on time series

Question

I am using the random forest for classifing if it will rain (1) or not (0) in my daily rain dataset with a small quantity of data (8103 tuples). Currently running a walking foward evaluation looking at the recall metric and getting a mean of 83.

I'm using a multivariate approach with the following columns:

I know the RF doesn't use autoregression, so I'm looking for ways to help it understand the time property better. Also if I there is any argument I can pass to RFClassifier that would be good, I'm already using bootstrap=False.

I already tried other techniques such as ETS and ARIMA, but they could not beat mean (only STLF has beaten it actually) so I'm looking into the Machine Learning approach.

score 1 · Accepted Answer · answered Jul 20 '22 at 17:55

1

You can always include lagged values of the target variable to account for autocorrelation. However, for a Boolean target, that will likely not add a lot of value. Also, much of the autoregressive behavior is probably already captured by the fact that your weather variables are already highly autoregressive (incidentally, you will need to forecast them, too).

You may want to consider Fourier transforms of day_of_year and wind_direction so your RF understands the circular nature of these predictors.

Are you training your RF on recall? If not, be aware it will likely optimize on a different loss function. Optimizing and evaluating on two different loss functions does not make a lot of sense (Kolassa, 2020), and recall is not a very good evaluation measure in any case.

answered Jul 20 '22 at 17:55

Stephan Kolassa

123,354

Hey, thanks for the response, I will try the transforms. As for the training, the RF is not being trained on recall, I'm using recall to evaluate how the model performs on the walk foward validation. How does one change the loss function on RF (I'm using sklearn)? The question you linked seems to suggest the accuracy is bad, and it really is even more on an unbalaced target such as rain, but from multiple sources I've come to the conclusion that recall on class 1 makes sense to evaluate rain. – Guilherme Jul 20 '22 at 18:08
Your paper seems to be focused on regression problems, I don't know if its possible to apply this line of thought for RF classification, the "loss functions" would be the "criterion" parameter if I'm not wrong, and the values it assumes are “gini”, “entropy”, “log_loss” – Guilherme Jul 20 '22 at 18:29
1

Unfortunately, I don't know how to change the loss function in the sklearn RF, sorry. The IMO best approach would be to aim for well-calibrated probabilistic predictions, then compare these to a threshold that is optimized for recall. (If you have found convincing sources that argue for recall, good for you. My experience is that most "data science" sources that argue for recall don't even understand the underlying statistical problem, much less address it.) ... – Stephan Kolassa Jul 20 '22 at 18:46
... Yes, my paper looks at numerical predictions, but I suspect the issue is precisely the same. Assume you have well-calibrated probabilistic predictions, and you look for one threshold that optimizes (say) expected recall, and another threshold that optimizes (say) expected precision, are the two thresholds the same? I haven't thought through this, but I would be surprised if they were. Thus, different loss functions imply different classifications if I am right about my hunch. Which is the classification analogue of what my paper looks at. – Stephan Kolassa Jul 20 '22 at 18:48
Ok, I will try those three metrics for the evaluation. Thank you very much for the responses! – Guilherme Jul 20 '22 at 19:00
1

Good luck! Incidentally, if you are only interested in recall, an easy way of getting a recall of 1 is to classify everything as the target class (essentially: comparing implicit probabilistic predictions to a threshold of 0). That is presumably not what you want, so you probably also want some precision or accuracy and use a larger threshold. Which is exactly my point: changing the accuracy measure changes the optimal output, even for precisely well-calibrated probabilistic predictions. You may be interested in this thread. – Stephan Kolassa Jul 21 '22 at 06:36

score 1 · Answer 2 · answered Jul 20 '22 at 18:56

1

You could give a dev package I wrote up a shot. It just wraps LightGBM (boosted trees) with some time series functionality such as fourier basis functions, 'custom' linear basis functions, and ar components.

Here is a quick example:

from sklearn.datasets import fetch_openml
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('darkgrid')
bike_sharing = fetch_openml("Bike_Sharing_Demand", version=2, as_frame=True)
y = bike_sharing.frame['count']
y = y[-800:].values
y_class = y > 70
y_class = y_class * 1
from LazyProphet import LazyProphet as lp
lp_model = lp.LazyProphet(seasonal_period=[24],
                      n_basis=10,
                      objective='classification',
                      fourier_order=5,
                      decay=.99,
                      ar=3,
                      return_proba=True)
fitted = lp_model.fit(y_class)
predicted_class = lp_model.predict(100)
plt.plot(y_class)
plt.plot(np.append(fitted, predicted_class), alpha=.5)
plt.axvline(800)
plt.show()

You can pass your own params such as the objective and all that through the 'boosting_params' argument.

And here is a multivariate example. It may be wonky so just let me know if there are issues.

answered Jul 20 '22 at 18:56

Tylerr

1,532

Thanks for the reponse, I'm trying to run the quick example you showed on collab but I'm getting ValueError: cannot convert float NaN to integer on fitted = lp_model.fit(y_class), but there is no NaN in y or y_class. Any ideas? – Guilherme Jul 20 '22 at 19:43
Probably a version issue with Pandas, what version are you running? Anymore detail of the traceback for error would help as well! – Tylerr Jul 20 '22 at 20:56
Traceback (most recent call last): File "create_lp_model.py", line 21, in <module> fitted = lp_model.fit(y_class) File "/.../lib/python3.8/site-packages/LazyProphet/LazyProphet.py", line 207, in fit self.X = self.build_input(self.scaled_y, X) File "/.../lib/python3.8/site-packages/LazyProphet/LazyProphet.py", line 174, in build_input shifted_y = LazyProphet.shift(shifted_y, ar_order) File "/...lib/python3.8/site-packages/LazyProphet/LazyProphet.py", line 142, in shift e[:n] = np.nan ValueError: cannot convert float NaN to integer Tried on pandas 1.3.5 and local on 1.4.2 – Guilherme Jul 20 '22 at 21:04
1

Awesome, I'll take a look. In the mean time it should work if you don't use ar components at all. – Tylerr Jul 20 '22 at 21:25
Hey, just ran here with my data and got a 0.28 RMSE, the best i had before was LSTM with 0.38. Would you recommend any resources to learning how this magic works e how to tune the parameters on the api? Thanks a lot! – Guilherme Jul 21 '22 at 01:55
1

I just debugged the issue, if you pass y as all ints with AR and classification it errors. The short fix should be to convert your y to floats before fitting. Fingers crossed it fixes the issue. – Tylerr Jul 21 '22 at 02:12
1

Good to hear! In terms of documentation, LightGBM itself has a ton of resources for the model itself. Most important things to tune are things like num_leaves and num_iterations. Most of the other stuff is just fancy feature engineering I got from GAMs: sin and cosine pairs for seasonality, and piecewise linear basis functions for trend. The weighting scheme for the plbf I think is novel so you are stuck with reading my tin foil conspiracies https://towardsdatascience.com/lazyprophet-time-series-forecasting-with-lightgbm-3745bafe5ce5. Feel free to reach out with anymore questions. – Tylerr Jul 21 '22 at 02:14
Hi @Tylerr, is it possible to use "predict_proba" with the package? – Guilherme Jul 24 '22 at 20:19
1

return_proba=True will return the probabilities, False with just do 1/0. – Tylerr Jul 24 '22 at 20:57

Fine-tune random forest on time series

2 Answers2