1

I was wondering whether it makes more sense to run Bayesian Optimization let's say ones with 100 iterations or ten times with ten iterations each. Which one should I favor?

I would assume that the first is better since the probabilistic model of the objective function gets better?

I'm using Bayesian Optimization to tune multiple hyperparameters of a Random Forest model - max_depth, max_features, min_samples_leaf, and min_samples_split. Currently, I'm using the library pyGPGO with a Gaussian Process and expected improvement as an acquisition function.

So S
  • 553
  • 5
  • 9

1 Answers1

2

We can't say without knowing your specific problem.

I would suggest you first run $10\times 10$. Then assess how variable your 10 results are (e.g., in terms of the variability of predictions on a holdout dataset). If this variability is "low", you can presumably trust 10 iterations. If this variability is "high", then I would not trust any one of the 10 iterations (I would probably trust an ensemble of the 10 instances more), and I would repeat with $10\times 100$, or whatever number is feasible, and iterate until the results stabilize.

You will need to consult your domain knowledge to operationalize "low" and "high".

Stephan Kolassa
  • 123,354
  • Thanks for the answer! The only problem, I'm asking about before actually running the experiments since each run is expensive - around 45 minutes of computation time. So I have to choose which way to go, before actually running the experiments :(. – So S Jul 20 '21 at 14:38
  • 1
    On the one hand, I would recommend trying this on a smaller sample or simulated data, so you get a feeling for how things stand. I would definitely trust that more than random strangers on the internet. On the other hand, if the only cost associated is computing time, you can either run it overnight, or parallelize it by renting a little cloud capacity. – Stephan Kolassa Jul 20 '21 at 14:41