Why does the model not overfit a single image?

[karkidaju]

Hi! I made some changes to the network for multiclass detection. As I tried to train it with a single image for a long time, I expected the loss to converge. But strangely, it first goes down to almost a zero value then spikes suddenly. Especially for regression it goes above the starting loss (but this is random, can be less too). I thought this was introduced by my changes. But then I reverted back to the original code and observed similar behavior.

This is the tensorboard log result with the original code. The model is being trained for 50 epochs on one image.

Is there a general concept in deep learning that I am missing? I thought theoretically the model would overfit the single image but the loss going up says otherwise.