Catheter action for RL #47
Comments
|
Hello, Your scene is probably similar to the CTR environment in SofaGym. You can look at the implementation here to help you, specifically the startCmd, start_cmd, and action_to_command functions. You don't need to explicitly implement the We will also add a new example of a catheter soon, based on the 3instrument scene from beamadapter, which is similar to your scene but with 3 concentric tubes instead of a single one, so keep an eye on this. |
|
Thank you so much for the help @samuelmyoussef . I now have something working but I still have a few questions. In the CTREnv.py in line 61 there is the variable dim_state and I'm not quite sure what it means. I understand why there are 12 actions in this example, but not what dim_state represents. Also it appears the goal is randomized, does this only happen when you start to train i.e. it is fixed for the entirety of each training attempt? Also do you happen to know what the variable xtip in the InterventionalRadiologyController represents? Again many thanks for the help. |
|
https://github.com/SofaDefrost/SofaGym/blob/main/sofagym/envs/CTR/CTRToolbox.py#L216 The goal is randomized for each episode not for the whole training attempt, so when an episode ends and the environment is reset, the goal will be different for the new episode. For |
|
Thanks for the explanations @samuelmyoussef . I now have something that trains well (it is quite a simple task). The catheter consistently gets to the goal position within the heart during training and is also using very few actions to get there. This sounds great however when I try to evaluate the trained agent, it just rotates at the start point which is not correct. Do you have any idea why the agent is very good at consistently hitting the maximum reward during training but then not when testing the trained agent? It might be useful to also mention that the simulation of the agent between epochs never hits the goal position either even though the agent is consistently hitting that same goal during the epoch. The goal i have set is the same across all epochs of training. If you have any ideas about this issue I would be grateful. Thanks once more! |
|
We are aware of this issue and it's been fixed in this PR #45 but it hasn't been merged to the main branch yet. Until then, you can try to use my branch from the PR or apply the modifications manually to see if it works and give us your feedback. |
|
Thanks for letting me know. When I try to run any RL training (even the examples) with this branch I get a bunch of errors. Now I have tried with this version from Aug 28, the RL runs however, it gets stuck at 99/1000 during epoch 1 even for the example environments. Please advise if I am doing something wrong. Thanks |
|
Can you post the errors log, please? |
|
Here is the error log for the up to date branch: `[ERROR] [SofaRuntime] AssertionError: The render_mode must be 'rgb_array', not None ########## SIG 15 - SIGTERM: a termination request was sent to the program ########## ########## SIG 15 - SIGTERM: a termination request was sent to the program ########## [ERROR] [SofaRuntime] ConnectionRefusedError: [Errno 111] Connection refused Exception ignored in: <function VecVideoRecorder.del at 0x7f9a5ff6ce50> |
|
Seems you are using a newer version of stable baseline 3, which doesn't work with SofaGym for now. Can you try with version 1.7.0 |
|
Thank you, this has fixed the errors. However, the training still stops at 99/1000 during epoch 1. There are no error messages. The environment simply resets every so often but remains on 99/1000. |
|
If you left the hyperparameters in the yaml file as they are, then it's running 5 evaluations every 100 steps, so it's probably just running this evaluation callback not stuck so it should continue after some time if you leave it running. You should change the eval_freq for whichever algorithm you're using if you want the evaluation to happen less frequently. |
|
I have left it running for longer now and I indeed do get an error after about 3 minutes. `[ERROR] [SofaRuntime] EOFError ########## SIG 15 - SIGTERM: a termination request was sent to the program ########## Exception ignored in: <function VecVideoRecorder.del at 0x7f34331bbb50> |
|
Is this error happening when you train your own environment or one of the examples? |
|
This is with the gripper-v0 environment, also happening with trunk-v0 Edit: My own environment and the gripper-v0 env both crash after the first 100 steps with the errors posted above. |
|
The trunk seems to be working right, the rendering windows are opened to record a video of the evaluation callback. Can you try again with the latest commit to the branch and see if it works. |
|
I have now tested it with the latest commit. The trunk env does work however, I still have the same issues with the gripper env and my own catheter env. |
|
I tested this issue more on the gripper environment. The error happens due to a timeout because the simulation stops responding and stops sending new observations after some time. In the case of the gripper, this is due to a wrong behavior in the simulation most probably because of very large deformation of the fingers causing it to crash. We will work on fixing this but for now If you limit the In the case of your own catheter env, I can't tell what's wrong with it but to help you debug, you could change the rendering during the callback to True in this line to have a better visualization of what's happening in the simulation during the evaluation. Line 294 in f94578f |
|
Thanks for the help. My own env crashes as soon as it starts to evaluate the trained model. This does not happen when I use the code from the main branch. This happens regardless of whether I have render set to True or False. I will paste the error bellow as it is different form the others i have posted thus far. Please let me know if you have any other ideas on how to fix. Thanks.
|
|
From the error log, it seems like a problem with the rendering because the image is not received correctly but I'm not really sure what's causing this. If you don't mind sharing the codes for your environment, it would make it easier for me to help you debug this issue. Can you also try to run your environment using the test script |
|
Sorry for the late reply, I am now trying on the new main sofagym repo. I still get the same error during the start of eval. It works on the test_env but it initially is in an odd state then corrects itself very quickly (I'm not sure if this is causing the issue). I have attached a simple version of my catheter environment. Thanks for the continued help. |
|
Additionally I am still working with the older version of sofagym (the one with broken evaluation). I am using the in built NatureCnn to process images as the input to a PPO. I am slightly confused by the overall network architecture (below). I don't understand the reason for the vf_features_extractor and pi_features_extractor layers. I realize that these are for the actor/critic however there are already features_extractor layers processing the image and it seems to me like it should only be one set of these layers. Are the pi/vf feature extractor layers necessary? Many thanks for the help.
|
Hi, I am trying to use sofa-gym in order to train an agent to move a catheter from one area of a model heart to another. Previously I have been using imitation learning techniques which have been relatively successful. My scene is based on this scene in the beam adapter example. This uses 'InterventionalRadiologyController' to move the catheter. Now I am following the CartPole example and am unsure how to implement the class 'ApplyAction' for my purposes, in order to move the catheter. Any help would be appreciated. Thanks, Mark.
The text was updated successfully, but these errors were encountered: