Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 25 vs previous value: 25. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize #87

Open
agarwalishan opened this issue Aug 4, 2020 · 0 comments

Comments

@agarwalishan
Copy link

I am doing NER using BERT for past months on google colab GPU and everything was working fine but now when I am doing same on CPUs I am getting this warning.
When am using colab GPU for training then there is no issue and no warnings of this kind. But when I am training with same data and same parameters I am getting this type of warning on running on colab CPU and on my local system's CPU and training never finishes.

WARNING:tensorflow:It seems that global step (tf.train.get_global_step) has not been increased. Current value (could be stable): 25 vs previous value: 25. You could increase the global step by passing tf.train.get_global_step() to Optimizer.apply_gradients or Optimizer.minimize.

Can someone please tell me the reason why problem occurs only on CPU not on GPU keeping the configurations same?
And how can I resolve it?
By the way BERT is using AdamOptimizer and I have never modified BERT's optimizer.py .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant