Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

train output loss diffurent from the test output loss #6

Open
Usernamezhx opened this issue Jun 8, 2017 · 5 comments
Open

train output loss diffurent from the test output loss #6

Usernamezhx opened this issue Jun 8, 2017 · 5 comments

Comments

@Usernamezhx
Copy link

hi
At first . thanks for your code . but I have a problem when i train with the ResNet_50_train_val.prototxt . Iteration 2w. Train output loss diffurent from the test output loss. the test loss was 2.0+ . what i have changed is the batch_size : I change it to 28 because of the memoray. others were same to your ResNet_50_train_val.prototxt and solver.prototxt . hope for your replying.

@antingshen
Copy link
Owner

Not sure I understand what your problem is

@Usernamezhx
Copy link
Author

thanks for your reply . i hope you can waste several minuts to read the log file .it is not long .
my_log.txt
i set the Test batch_size : 1 in train_val.prototxt . so if i want to test all my test sample . i have to set the test_iter: 100000. because my test sample is 100000 .it is so big that i will take 1 hour to finish it.
it is wrong ?
hope your reply

@antingshen
Copy link
Owner

Your train loss isn't going down, so your net isn't learning. I'd hold on the testing until training is working.

I can't help you figure out why your net isn't learning, but one thing to look at is to make sure that you increased your iter_size when decreasing batch size to ensure the same mathematical batch size for updates.

@qdinfish
Copy link

qdinfish commented Jan 9, 2018

@Usernamezhx Hi, I encounter the same issue, have you fund the root cause ?

@ghost
Copy link

ghost commented Oct 9, 2018

@Usernamezhx your training loss is way too small here, I'd guess that your training accuracy is pretty high while testing is somewhat lower than expected. Are you using ImageNet here? One of the issues could be the dataset leading to the network overfitting.

Were you able to solve this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants