CNN model accuracy is increasing really slowly

Question

I am training a CNN model from scratch on the Caltech101 dataset. The accuracy of the model is increasing very slowly after the 5th epoch. Shown below is the accuracy and loss curves of the model for 50 epochs.

The full training log is found here: https://pastebin.com/P5VJTeK2

This model has attained a 90% accuracy score on a smaller dataset with 700+ images after 50 epoch and I do not know why the model is unable to learn features on the caltech101 dataset. Overfitting is definitely not the reason here as the training and validation curves are still really close to each other. I just want to know why this is happening, it is a plus if you have solutions to this problem too.

Data preprocessing:

train_datagen = ImageDataGenerator(dtype = 'float32', 
                                   rotation_range = 20,
                                   width_shift_range=0.2,
                                   height_shift_range=0.2,
                                   horizontal_flip=True,
                                  validation_split = 0.2)
test_datagen = ImageDataGenerator(dtype = 'float32', 
            validation_split = 0.2)
train_generator = train_datagen.flow_from_directory('data/caltech101', batch_size = 8, class_mode = 'categorical', target_size = (224, 224), subset = 'training')
validation_generator = test_datagen.flow_from_directory( 'data/caltech101', batch_size = 8, class_mode = 'categorical', target_size = (224, 224), subset = 'validation')

Model architecture:

model_one = Sequential([
    tf.keras.layers.experimental.preprocessing.Rescaling(1./255),
    Conv2D(filters=64,kernel_size=(4,4), activation='relu'),
    Conv2D(filters=32,kernel_size=(3,3), activation='relu'),
    AveragePooling2D(pool_size=(2,2)),
Conv2D(filters=32,kernel_size=(3,3), activation='relu'),
Conv2D(filters=32,kernel_size=(3,3), activation='relu'),
Conv2D(filters=32,kernel_size=(3,3), activation='relu'),
AveragePooling2D(pool_size=(2,2)),
Flatten(),

Dense(256, activation='relu'),
Dropout(0.2),
Dense(256, activation='relu'),
Dropout(0.2),
Dense(128, activation='relu'),
Dropout(0.2),
Dense(128, activation='tanh'),
Dropout(0.2),
Dense(102, activation='softmax')


])
model_one.compile(tf.keras.optimizers.RMSprop(learning_rate = 0.001),
                  loss = 'categorical_crossentropy',
              metrics = ['accuracy'])