Random Forest - varying seed to quantify uncertainty

Question

I would like to quantify the uncertainty of a Random Forest binary classifier.

The idea that popped in my mind was to fit the Random Forest 100 times with different seeds. Computing the variance of the forest's output on data in a validation set, I can then estimate the variance of the model.

Notice that the technique is very similar to what is done with neural network, where we randomly initialize different models and use the ensemble of nets to estimate the variance of the single net.

Do you see some flaws in this procedure?

Thank you in advance!

---

Disclaimer: I am aware of different approaches to estimate the variance of a Random Forest, and of some implementations in Python. However, if the number of samples is high, such estimator require a very high number of trees in the forest in order to converge. In my dataset, I am dealing with ~500k data points and 100 trees. For this reason, using such estimation techniques is not possible.

---

I report here a reproducible example that shows a situation where the aforementioned available estimator does not converge:

import numpy as np import sklearn.datasets from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split import forestci as fci

X, y = sklearn.datasets.make_blobs(n_samples=100000, n_features=5, centers=2)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
clf = RandomForestClassifier(random_state=61, n_estimators=100, n_jobs = 10, criterion='entropy')
clf.fit(X_train, y_train)
y_test_var = fci.random_forest_error(clf, X_train=X_train, X_test=X_test)
max(y_test_var) # should be <0.25, since we are dealing with binary data, but isn't

Answer 1

Why would you do that? As you noticed, there already exists a solution by respected authors including Trevor Hastie and Bradley Efron who rigorously studied and empirically validated the solution. If you are planning to introduce an alternative solution, you should probably start with equally rigorous mathematical proofs and an empirical evaluation vs their solution. It sounds like an interesting research paper, but if you want to use it as an applied method, re-inventing the wheel is probably not the best idea.

Your proposed method considers the random seed as the only source of randomness. It does not take into account the randomness in the data due to sampling. If you are intending to use it for calculating confidence intervals it would underestimate the intervals because of that. Same argument would apply to varying the random seed only for neural networks. You need more than this, for example, use random samples of the training data (bootstrap) for each in the models in ensemble used for estimating the variability, and for neural network-specific approaches things like Monte Carlo dropout, or injecting randomness to the weights, etc.