stochastic gradient descent regression, linear model, R^2 drops suddenly

Question

I 'm trying to use a batch gradient descent algorithm to do linear regression on a large dataset- I load up as much data as my computer can handle, do a partial fit, print out some diagnostics to a CSV, and repeat

python pseudocode

m_scaler = sklearn.preprocessing.StandardScaler()
m_data = <3600*100*24*9 samples, with 15 X channels, and one y channel>
sgdR_01 = SGDRegressor(n_iter=m_iter,alpha = 10.0**-3)
i=0
while i < 100:
    i=i+1
    df = <select_360000_samples from m_data>
    df = <preprocess>
    m_scaler.partialFit(df[x_channels])
    df[x_channels] =  m_scaler.transform(df[x_channels])
    X_train,y_train,X_test,y_test = trainTestSplit(df)
    sgdR_01.partial_fit(X_train,y_train)
    <track train/test score, train/test MSE, and coefficients for sgdR_01>



preprocessing steps:
    add polynomial combinations of certain channels
    oversample so y has a 'flat histogram'
    randomly select ~ 200000 samples so my computer can handle the data

Right now, I'm hovering around .65 for my R^2 score, but every few iterations my score will drop to like -50 or -900. At the next iteration, it'll be back around .65 What's going on when that happens? Why is SGDR so erratic?

Getting a negative MSE has to be a bug. The lowest possible value of $x^2$ is $0$ assuming you are predicting using real numbers . . . — Neil Slater, Aug 01 '17 at 15:42
@NeilSlater from: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDRegressor.html#sklearn.linear_model.SGDRegressor.score it just means it's arbitrarily bad. My question focuses more on the fact that it drops suddenly, not on what exactly it drops to — Mohammad Athar, Aug 01 '17 at 15:46
Sorry, I saw MSE in your pseudocode, did not notice you were using that score. — Neil Slater, Aug 01 '17 at 15:51
Corroborating the scikit documentation, When is R squared negative? — Firebug, Aug 01 '17 at 16:10

stochastic gradient descent regression, linear model, R^2 drops suddenly

0 Answers0