My ANN model produces classic overfitting characteristics, producing high R2 values (90-99%) but low accuracy scores (10-40%). I'm currently inputting 28655 data entries, using 8 input features to predict solar radiation.
My model is as follows:
df = pd.read_csv (r'D:\ABC Masters Diss\Penarth 1 year.csv')
print (df)
df[:5]
df.info()
#Features
X = df[["Cloud Coverage", "Probability of Precipitation", "Dry Bulb Temp", "Dew Point Temp", "Relative Humidity", "Apparent Temp", "Time of Day", "Month"]]
#Target
Y = df["Solar Radiation"]
X_train, X_test, Y_train, Y_test = train_test_split(X,
Y,
test_size = 0.2)
scaler = StandardScaler()
X_train_std = scaler.fit_transform(X_train)
X_test_std = scaler.transform(X_test)
#Initialise the ANN
model = Sequential()
#Input layer and hidden layer
model.add(Dense(50, activation= 'relu', input_dim = 8))
#second layer
model.add(Dense(units = 50, activation = 'relu'))
#output layer
model.add(Dense(units = 1))
model.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics = ['accuracy'])
model.fit(X_train_std, Y_train, batch_size = 10, epochs = 10)
pred_a1 = model.predict(X_test_std)
pd.DataFrame({"Actual": Y_test, "Predicted": pred_a1.flatten()})[:5]
I've tried altering the architecture, no. of layers & neurons etc. I've also tried changing the activation, adding drop out and kernel constraints and altering the learning rate(see below). Any ideas on how to improve this?
#Input layer and hidden layer
model.add(Dense(50, activation= 'relu', input_dim = 8, kernel_constraint=MaxNorm(3)))
model.add(Dropout(0.2))
#second layer
model.add(Dense(units = 50, activation = 'relu', kernel_constraint=MaxNorm(3)))
model.add(Dropout(0.2))
#output layer
model.add(Dense(units = 1, activation = 'relu'))
sgd = SGD(learning_rate=0.01, momentum=0.9)
model.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics = ['accuracy'])
model.fit(X_train_std, Y_train, batch_size = 10, epochs = 100)
train_test_splitassumes they are, but your data doesn't look so... if there is temporary correlation between rows, then train_test_split can't be used. That said, try a simpler model. Reduce the number of neurons and layers. Drop out may also help. Also include the loss function wile training for both, train and validation to see how it evolves. – Rodrigo Laguna Aug 18 '22 at 21:21