Is there any method to generate image state?

[SangbaekYoo]

Hi, thank you for great works for SofaGym!

I currently study SofaGym and make my own environment.
In my reinforcement learning architecture, I want to define image state(or observation) which is related to GUI screen image.

For this, I defined additional camera in Scene python file as follows:
(I thought that InteractiveCamera can realize my wish :))

source = config["source"]
target = config["target"]
rootNode.addObject("InteractiveCamera", name='camera2', position=source, lookAt=target, zFar=500)

Next, I refer to viewer.py and find the code which can make image object as follows:

glViewport(0, 0, self.surface_size[0], self.surface_size[1])

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
glEnable(GL_LIGHTING)
glEnable(GL_DEPTH_TEST)

if self.root:
    Sofa.SofaGL.glewInit()
    Sofa.Simulation.initVisual(self.root)
    glMatrixMode(GL_PROJECTION)
    glLoadIdentity()
    gluPerspective(45, (self.surface_size[0] / self.surface_size[1]), 0.1, self.zFar)

    glMatrixMode(GL_MODELVIEW)
    glLoadIdentity()

    cameraMVM = self.root.camera.getOpenGLModelViewMatrix()

    glMultMatrixd(cameraMVM)
    Sofa.SofaGL.draw(self.root)
else:
    print("===============> ERROR")

try:
    x, y, width, height = glGetIntegerv(GL_VIEWPORT)
except:
    width, height = self.surface_size[0], self.surface_size[1]
buff = glReadPixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE)

image_array = np.fromstring(buff, np.uint8)
if image_array.shape != (0,):
    image = image_array.reshape(self.surface_size[1], self.surface_size[0], 3)
else:
    image = np.zeros((self.surface_size[1], self.surface_size[0], 3))

image = np.flipud(image)
image = np.moveaxis(image, 0, 1)

I checked that above image is same as GUI screen image.
By using above code and Interactive camera(defined as camera2), I implemented additional function that returns image in Toolbox python file.
However, there is some multi-thread error(I guess that problem is due to reinitialization such as Sofa.SofaGL.glewInit() and Sofa.SofaGL.draw(self.root)).
So I conclude that this is not good approach.

Is there any method to get screen image in Toolbox and Scene code python without GUI?

[alxbilger]

@damienmarchal Have you done such things?

[damienmarchal]

Hi @SangbaekYoo, @alxbilger

I didn't tried it within SofaGym but I think that the approach to make rendering in Sofa is to use this plugin https://github.com/jnbrunet/SofaOffscreenCamera

It may requires some work to make it work within SofaGym but I see no reason why it shouldn't be possible.

[SangbaekYoo]

I choose config parameter 'save_image' to True, and read written image in reward function. I think that render function in viewer.py is necessary to generate image and since I saw that visualization module and simulation module do not work in same process, file read/write approach is only way to get image.

Do you also agree this approach?

[SangbaekYoo]

@damienmarchal

I also approach this method, but there is some build error.
Below link is my recent problem.
sofa-framework/sofa#4146 (comment)

[samuelmyoussef]

Hello @SangbaekYoo,

Maybe you could try to override the step method in your environment's class and call the render method inside it after applying the action to get the image and return it as the obs.

If you also want to train this environment using stable baselines, you will need to modify the policy hyperparameter in the YAML file to CnnPolicy instead of MlpPolicy.

[SangbaekYoo]

@samuelmyoussef Thank you for great advice, i will try your policy suggestion.

I saw that add_new_step function in AbstractEnv.py does action and calculates reward and gets state.(action->reward->next_state)
So I implemented new functions and splited as (action) and (reward->next_state), and inserted render method to between them. Is this similar with your suggestion?

[samuelmyoussef]

Yes, your implementation is similar to what I'm suggesting but it could be easier since you can just override your step method in your environment to call the parent step method from AbstractEnv to do the action and get the reward and state, then discard the state you got and replace it with the image you get by calling the render method. I feel it's not the best way to do it but the most straightforward for now.

[SangbaekYoo]

Hello @samuelmyoussef, and thank you for sharing various approaches.

I see that if I define observation or state by big size image, since there is process to change (dictionary class <-> string), consumed time is very inefficient. So I decide to save image in Toolbox.py and read in step method.

[samuelmyoussef]

I don't think it will make much difference whether it's saving the image from the Toolbox or from step. If you want something that works then you can implement whichever approach is more straightforward for you. However, if you're looking for the most efficient way then both methods are not that good since you're still relying on the viewer to render the scene and save the image, which is really slow and not efficient at all, but it's the only way in SofaGym for now.

To have a more efficient approach, you should look into what Damien linked about the offscreen camera since you won't have to launch the GUI just to save a state image.

[ScheiklP]

I have a pretty efficient solution in https://github.com/ScheiklP/sofa_env that does not require the GUI or SofaOffscreenCamera.

I also have a rudimentary solution of multiple cameras in the same scene, but that did not yet make it into the upstream version of the repo.

Maybe that could help?

[SangbaekYoo]

@ScheiklP What code can I reference?

[ScheiklP]

@SangbaekYoo
The base class contains all you need. https://github.com/ScheiklP/sofa_env/blob/main/sofa_env/base.py

• For generating the rendering context: https://github.com/ScheiklP/sofa_env/blob/main/sofa_env/base.py#L374
• For rendering: https://github.com/ScheiklP/sofa_env/blob/main/sofa_env/base.py#L402

The camera is defined in the scene description.
https://github.com/ScheiklP/sofa_env/blob/main/sofa_env/scenes/controllable_object_example/scene_description.py

[MarkWrobelUCL]

Hi, I am trying to do something similar. I want to use an image (or set of images) as my state. I am currently using just some coordinate points. In contrast to the original post I am happy with constructing the images myself and do not need them to be the same as the GUI. In order to make this change I understand I need to pass the image as my state and change the policy hyperparemeter to CnnPolicy instead of MlpPolicy. I assume I also need to somehow change my observation space in my env.py file to match the image I am passing as my state but I am unsure exactly how to do this? Is there anything else that I need to change in order to get this to work?

[samuelmyoussef]

your observation space will be something like this observation_space=Box(0, 255, (image_height, image_width, channels), np.uint8) where the channels will be 3 if it's an rgb image for example.

Other than this, you need to do the changes to make sure that your step and reset methods are correctly returning the observation as your image state.

To use a set of images instead of one, I think there's a frame stack wrapper already implemented in gym that you can use to do so.

[MarkWrobelUCL]

I have the model training well with the images but I am unsure how to load the model once it is trained. When training a model with coordinate points i just use PPO.load() as I am using the PPO algo. However, with a customCNN passed into the PPO during training, PPO.load() no longer works to load the trained model. Is there another way of loading a trained model with a customCNN?

[samuelmyoussef]

Can you provide more details about the issue? What happens when you try to use PPO.load()? If it causes an error, please share the error log here.

[MarkWrobelUCL]

This is the error I get when I try use PPO.load() to load the trained model with the customCNN:

[ERROR] [PythonScript] ModuleNotFoundError: No module named 'agents' File "/home/mark/Documents/SOFA/PA catheterization simulation/PA_cath_sim_4_controlled_rl_xyz.py", line 17, in <module> model = PPO.load('./models/rl/3000.zip') File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/base_class.py", line 679, in load data, params, pytorch_variables = load_from_zip_file( File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/save_util.py", line 421, in load_from_zip_file data = json_to_data(json_data, custom_objects=custom_objects) File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/save_util.py", line 164, in json_to_data deserialized_object = cloudpickle.loads(base64_object)

[SangbaekYoo]

@MarkWrobelUCL
Did you run your code in different virtual environment? I had similar problem when I evaluated and run trained model in other environment because I did not install same modules.

[MarkWrobelUCL]

I actually just fixed my issue and that was not the problem. I needed to import my CustomCNN class before loading the model. This class was defined in the 'agents' folder.