I want to do a convergence study of Ridge regression with increasing training size. For this I train the Ridge regressor using hyperopt for different training sizes [10, 20, 40, 80]. As far as I understand, the TPE strategy adapts to the optimization, so I would expect that the sampling for each training with different training sizes be different, but that's not the case. Here is a code that shows that TPE is always sampling the same points:
import numpy as np
X = np.random.randn(100,1)
Y = np.random.randn(100,1)
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y,
test_size=0.2,
random_state=0)
from sklearn.linear_model import Ridge
from hyperopt import fmin, tpe, hp, Trials
results = []
for n in [10, 20, 40, 80]:
print "\nTraining with", n, "samples"
def objective(alpha):
model = Ridge(alpha=alpha).fit(X_train[:n], Y_train[:n])
return -model.score(X_test, Y_test) # negative so that lower means better
trials = Trials()
best = fmin(objective, hp.uniform('alpha', 0.0, 10.0),
algo=tpe.suggest,
max_evals=5,
trials=trials)
print "alphas sampled:"
for trial in trials:
print trial['misc']['vals']
The output of this is:
Training with 10 samples
alphas tried:
{'alpha': [6.964691855978616]}
{'alpha': [1.0606490595473272]}
{'alpha': [5.067260114995699]}
{'alpha': [1.0517212721767522]}
{'alpha': [5.238333199546759]}
Training with 20 samples
alphas tried:
{'alpha': [6.964691855978616]}
{'alpha': [1.0606490595473272]}
{'alpha': [5.067260114995699]}
{'alpha': [1.0517212721767522]}
{'alpha': [5.238333199546759]}
Training with 40 samples
alphas tried:
{'alpha': [6.964691855978616]}
{'alpha': [1.0606490595473272]}
{'alpha': [5.067260114995699]}
{'alpha': [1.0517212721767522]}
{'alpha': [5.238333199546759]}
Training with 80 samples
alphas tried:
{'alpha': [6.964691855978616]}
{'alpha': [1.0606490595473272]}
{'alpha': [5.067260114995699]}
{'alpha': [1.0517212721767522]}
{'alpha': [5.238333199546759]}