Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to validate at training start of deepedit #515

Merged
merged 11 commits into from
Oct 11, 2023
Merged

Add option to validate at training start of deepedit #515

merged 11 commits into from
Oct 11, 2023

Conversation

mingxin-zheng
Copy link
Contributor

Description

Modify bundle as the new feature enabled in Project-MONAI/MONAI@fc1350a

Status

Ready/Work in progress/Hold

Please ensure all the checkboxes:

  • Codeformat tests passed locally by running ./runtests.sh --codeformat.
  • In-line docstrings updated.
  • Update version and changelog in metadata.json if changing an existing bundle.
  • Please ensure the naming rules in config files meet our requirements (please refer to: CONTRIBUTING.md).
  • Ensure versions of packages such as monai, pytorch and numpy are correct in metadata.json.
  • Descriptions should be consistent with the content, such as eval_metrics of the provided weights and TorchScript modules.
  • Files larger than 25MB are excluded and replaced by providing download links in large_file.yml.
  • Avoid using path that contains personal information within config files (such as use /home/your_name/ for "bundle_root").

@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv
Copy link
Collaborator

need to use new feature in MONAI dev branch, thus it requires a new monai version (currently 1.2.0 shown in https://github.com/Project-MONAI/model-zoo/pull/515/files#diff-90cacb33aaa1f750be54f039f8d7b09347e90155d8293c52c4fd7c3f79420575L30).
We may have to use a new tag cc @wyli

Signed-off-by: Mingxin Zheng <[email protected]>
@mingxin-zheng
Copy link
Contributor Author

I hacked the ci install dev branch of monai for testing purposes 8484dbd. That may not be something interesting to keep, but just to get an idea if the bundle can pass test.

Signed-off-by: Mingxin Zheng <[email protected]>
Signed-off-by: Mingxin Zheng <[email protected]>
Signed-off-by: Mingxin Zheng <[email protected]>
Signed-off-by: Mingxin Zheng <[email protected]>
@mingxin-zheng
Copy link
Contributor Author

After discussion with @yiheng-wang-nv , I reverted and will wait for a new rc tagging,

@mingxin-zheng mingxin-zheng requested a review from Nic-Ma October 11, 2023 05:23
@wyli
Copy link
Collaborator

wyli commented Oct 11, 2023

sure, I'll tag another rc today to test the latest feature

Signed-off-by: Mingxin Zheng <[email protected]>
@wyli
Copy link
Collaborator

wyli commented Oct 11, 2023

/build

1 similar comment
@wyli
Copy link
Collaborator

wyli commented Oct 11, 2023

/build

@wyli
Copy link
Collaborator

wyli commented Oct 11, 2023

some errors testing with rc4, probably due to #517:

[2023-10-11T08:13:25.441Z] 2023-10-11 08:13:25,380 - ignite.engine.engine.SupervisedEvaluator - INFO - Got new best metric of val_mean_dice: 0.5179497599601746
[2023-10-11T08:13:25.441Z] 2023-10-11 08:13:25,380 - INFO - Epoch[1] Metrics -- val_mean_dice: 0.5179 
[2023-10-11T08:13:25.441Z] 2023-10-11 08:13:25,380 - INFO - Key metric: val_mean_dice best value: 0.5179497599601746 at epoch: 1
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,783 - ignite.engine.engine.SupervisedEvaluator - INFO - Epoch[1] Complete. Time taken: 00:00:01
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,784 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run complete. Time taken: 00:00:01
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,784 - INFO - Epoch[1] Metrics -- train_dice: 0.5166 
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,784 - INFO - Key metric: train_dice best value: 0.5165802836418152 at epoch: 1
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,785 - ignite.engine.engine.SupervisedTrainer - INFO - Epoch[1] Complete. Time taken: 00:00:02
[2023-10-11T08:13:26.004Z] 2023-10-11 08:13:25,785 - ignite.engine.engine.SupervisedTrainer - INFO - Engine run complete. Time taken: 00:00:02
[2023-10-11T08:13:26.004Z] there are more than 1 component have name `Interaction`: ['monai.apps.deepedit.interaction', 'monai.apps.deepgrow.interaction'], use the first one `monai.apps.deepedit.interaction. if want to use others, please set its full module path in `_target_` directly.
[2023-10-11T08:13:27.370Z] .2023-10-11 08:13:27,138 - INFO - Setting logging properties based on config: models/spleen_deepedit_annotation/configs/logging.conf.
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,791 - ignite.engine.engine.SupervisedEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,853 - ignite.engine.engine.SupervisedEvaluator - INFO - Restored all variables from models/spleen_deepedit_annotation/models/model.pt
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,926 - INFO - Number of simulated clicks: 2
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,951 - ignite.engine.engine.SupervisedEvaluator - ERROR - Current run is terminating due to exception: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,951 - ERROR - Exception: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] Traceback (most recent call last):
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 841, in _run_once_on_dataset
[2023-10-11T08:13:28.005Z]     self._fire_event(Events.ITERATION_COMPLETED)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 421, in _fire_event
[2023-10-11T08:13:28.005Z]     func(*first, *(event_args + others), **kwargs)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/metrics_saver.py", line 124, in _get_filenames
[2023-10-11T08:13:28.005Z]     meta_data = self.batch_transform(engine.state.batch)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in _wrapper
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in <listcomp>
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in <listcomp>
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z] KeyError: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,951 - ignite.engine.engine.SupervisedEvaluator - ERROR - Engine run is terminating due to exception: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] 2023-10-11 08:13:27,951 - ERROR - Exception: 'image_meta_dict'
[2023-10-11T08:13:28.005Z] Traceback (most recent call last):
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 753, in _internal_run
[2023-10-11T08:13:28.005Z]     time_taken = self._run_once_on_dataset()
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 854, in _run_once_on_dataset
[2023-10-11T08:13:28.005Z]     self._handle_exception(e)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 464, in _handle_exception
[2023-10-11T08:13:28.005Z]     self._fire_event(Events.EXCEPTION_RAISED, e)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 421, in _fire_event
[2023-10-11T08:13:28.005Z]     func(*first, *(event_args + others), **kwargs)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/stats_handler.py", line 202, in exception_raised
[2023-10-11T08:13:28.005Z]     raise e
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 841, in _run_once_on_dataset
[2023-10-11T08:13:28.005Z]     self._fire_event(Events.ITERATION_COMPLETED)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/ignite/engine/engine.py", line 421, in _fire_event
[2023-10-11T08:13:28.005Z]     func(*first, *(event_args + others), **kwargs)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/metrics_saver.py", line 124, in _get_filenames
[2023-10-11T08:13:28.005Z]     meta_data = self.batch_transform(engine.state.batch)
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in _wrapper
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in <listcomp>
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z]   File "/root/.local/share/virtualenvs/MONAI-zoo-premerge-CHR6GQKS/lib/python3.8/site-packages/monai/handlers/utils.py", line 199, in <listcomp>
[2023-10-11T08:13:28.005Z]     ret = [data[0][k] if first else [i[k] for i in data] for k in _keys]
[2023-10-11T08:13:28.005Z] KeyError: 'image_meta_dict'

@wyli
Copy link
Collaborator

wyli commented Oct 11, 2023

possible fix Screenshot 2023-10-11 at 09 43 14

Signed-off-by: KumoLiu <[email protected]>
@wyli
Copy link
Collaborator

wyli commented Oct 11, 2023

/build

Signed-off-by: Mingxin Zheng <[email protected]>
@mingxin-zheng
Copy link
Contributor Author

/build

2 similar comments
@yiheng-wang-nv
Copy link
Collaborator

/build

@wyli
Copy link
Collaborator

wyli commented Oct 11, 2023

/build

@wyli wyli merged commit 28926e2 into dev Oct 11, 2023
@wyli wyli deleted the eval_start branch October 11, 2023 15:39
yiheng-wang-nv pushed a commit to yiheng-wang-nv/model-zoo that referenced this pull request Jul 29, 2024
### Description

Modify bundle as the new feature enabled in
Project-MONAI/MONAI@fc1350a

### Status
**Ready/Work in progress/Hold**

### Please ensure all the checkboxes:
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Codeformat tests passed locally by running `./runtests.sh
--codeformat`.
- [ ] In-line docstrings updated.
- [ ] Update `version` and `changelog` in `metadata.json` if changing an
existing bundle.
- [ ] Please ensure the naming rules in config files meet our
requirements (please refer to: `CONTRIBUTING.md`).
- [ ] Ensure versions of packages such as `monai`, `pytorch` and `numpy`
are correct in `metadata.json`.
- [ ] Descriptions should be consistent with the content, such as
`eval_metrics` of the provided weights and TorchScript modules.
- [ ] Files larger than 25MB are excluded and replaced by providing
download links in `large_file.yml`.
- [ ] Avoid using path that contains personal information within config
files (such as use `/home/your_name/` for `"bundle_root"`).

---------

Signed-off-by: Mingxin Zheng <[email protected]>
Signed-off-by: KumoLiu <[email protected]>
Co-authored-by: KumoLiu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants