Problem Description
When training a prediction model using Keras, the loss and accuracy sometimes get stuck. For example:
Epoch 1/10
2499007/2499007 [==============================] - 2s - loss: 1.9112 - acc: 0.5325
Epoch 2/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 3/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 4/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 5/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 6/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 7/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 8/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 9/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 10/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
This issue could be caused by several reasons. Here we summarize some solutions to fix it.
Solutions
Changing Optimizer
A list of optimizers in Keras: https://keras.io/optimizers/
Adjusting Learning Rate
Tune the network by adjusting learning rate of the optimizer. For example, increase learning from 0.00001 to 0.1:
sgd = optimizers.SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
Normalizing Activations
Add BatchNormalization after each layer:
...
model.add(Dense(20, activation='relu'))
model.add(BatchNormalization())
...
References
Loss not changing when training
Keras accuracy does not change