waypoint_type etc. None in waypoints_info in cliport_test

[le-horizon]

Hi VLMBench gurus,

I'm just trying to run the cliport_test script with the provided models.

I installed the code and dependencies, and downloaded the trained model, and the seen and unseen splits of data:
bash download_dataset.sh -s ~/vlmbench/Dataset -p valid_unseen -t pick
bash download_dataset.sh -s ~/vlmbench/Dataset -p valid_seen -t pick

When running cliport_test, waypoint_type etc. fields are all None in waypoints_info array.
python vlm/scripts/cliport_test.py --task pick --data_folder /Path/Dataset/valid --checkpoints_folder /Path/models

This is the waypoints_info array:
['waypoint0', None, None, 1, None, False, array([ 0.09891216, -0.09790423, 0.85907155, -0.64400905, 0.76425868,
-0.01948901, 0.02795067])], ['waypoint1', None, None, 1, None, False, array([ 0.10433818, -0.0974073 , 0.7792573 , -0.64400905, 0.76425868,
-0.01948901, 0.02795067])], ['waypoint2', None, None, 1, None, False, array([ 0.09755566, -0.09802846, 0.8790251 , -0.64400905, 0.76425868,
-0.01948901, 0.02795067])], ['waypoint3', None, None, 1, None, False, array([ 0.43130848, -0.15798603, 0.85353625, 0.39130512, 0.79267031,
-0.0298808 , 0.46654186])]

The code catches this error and says "need re-generate: /Path/valid/seen/pick_cube_shape/variation0/episodes/episode4".

Am I missing some installation step? Am I downloading the correct dataset files?

Thanks,
Le

[KzZheng]

Thanks for reaching out. The None parts of waypoint information are used for sub-step evaluations, and we do not save this information in the current validation sets (The validation sets were generated previously and they are time-consuming due to the rendering). Therefore, there are two ways to fix this:

1. Testing on the test sets or re-generating the validation sets. We save this information in the test sets and the newest generation code also saves this information for the training/validation generations.
2. If you do not need to evaluate the sub-goal success, you can comment out the related code in the test python file, and it can still work on evaluating the whole task's success.

[le-horizon]

Thanks for the detailed response @KzZheng !
I simply skipped all checks on action_type or waypoint_type when they are None, e.g. "if action_type is not None and "grasp" in action_type".

However, validation is showing 0% success. And it sometimes says "Could not create path."

Do you know what might be off?

Thanks,
Le

[KzZheng]

I re-download the unseen validation set for picking, and I found I could run the test on this task without any code change. The results are also not zeros. Can you check whether you are using the correct weights and dataset? Thanks!

PS: "Could not create path" is normal since the estimation errors will cause the failures of path planning. You can try to set "--ignore_collision" as True to ignore the collisions.

[KzZheng]

The issues are fixed by copying the Lua file into the Coppliasim folder.

[le-horizon]

waypoint_type missing (code complains need re-generate) turns out to be because I didn't copy over the lua script.
Thanks for the help Kaizhi!

There is also a gdrive download issue tracked here: prasmussen/gdrive#645
rclone (rclone-v1.59.2-linux-amd64) is able to solve the problem, just a bit of extra setup needed before copying - need to create a shortcut of the source google drive directory in your own drive.

[daichisaito-cs]

@le-horizon @KzZheng

Hi, I'm also tying to get this code to work, but how did you manage it?
I tried three ways to fix them:

1. skipping all checks on action_type or waypoint_type when they are None, e.g. "if action_type is not None and "grasp" in action_type".
2. Testing on the test sets
3. Re-generating the validation sets and testing on the validation sets
   (In Solution 1, the robot was moving but did not grasp anything)

Are the test data I downloaded wrong?

Thanks,
Daichi

[KzZheng]

Hi Daichi @daichisaito-cs,

Le fixes the error by copying the Lua file (https://github.com/eric-ai-lab/VLMbench/blob/main/simAddOnScript_PyRep.lua) into the Coppliasim folder. I'm not sure whether you have the same issues.

Best,
Kaizhi

[daichisaito-cs]

Thanks for quick reply,

I tried copying that file into the Coppliasim folder, but it didn't work and I got the error 'need re-generate ~'.
I'm just wondering why it didn't work on the test sets either. Do you have any ideas?

[KzZheng]

There are only two potential issues that will lead to 'need re-generate' in my mind:

1. Wrong data folder. Make sure all test data are correctly loaded.
2. Lua file. Each time you re-install PyRep, this file needs to be copied again.

I think you should be one of these cases.

[KzZheng]

You can also add

import traceback
print(traceback.format_exc())

Under the exception

VLMbench/vlm/scripts/cliport_test.py

Line 329 in dfc3e53

except:

to provide more error details.

[daichisaito-cs]

I already copied lua file, so it might be the first issue you mentioned.

Is the following link right for downloading the test_seen data?
https://drive.google.com/drive/folders/1tuGIlRm_0xUh1WZFlYjJmcN0QX6sf6Nl

[KzZheng]

The link is correct. Can you confirm you use the Coppeliasim 4.1?

[daichisaito-cs]

Yes, I use the Coppeliasim 4.1.0.

[KzZheng]

You can also add

import traceback
print(traceback.format_exc())

Under the exception

VLMbench/vlm/scripts/cliport_test.py

Line 329 in dfc3e53

except:

to provide more error details.

Can you add this code to the file to provide more error details?

[daichisaito-cs]

The error I got when I added traceback was like this:

Traceback (most recent call last):
  File "vlm/scripts/cliport_test.py", line 314, in <module>
    if len(waypoints_info['waypoint1']['target_obj_name'])!=0:
TypeError: object of type 'NoneType' has no len()

[KzZheng]

I just test the code. Every thing works fine for me. Can you add the following code block

from amsolver.backend.utils import ReadCustomDataBlock, WriteCustomDataBlock
dummy = task._task.task_base
WriteCustomDataBlock(dummy.get_handle(),"test_a","test_b")
b = ReadCustomDataBlock(dummy.get_handle(), "test_a")
print(b)

before

VLMbench/vlm/scripts/cliport_test.py

Line 306 in dfc3e53

waypoints_info = {name: obj for name, obj in obs.object_informations.items() if "waypoint" in name}

To see whether you can successfully write and read metadata from coppeliasim?

[daichisaito-cs]

When I run the picking task, I got the following error.

rkspace/VLMbench/dataset/test --checkpoints_folder ./models                      
Attn FCN - Stream One: plain_resnet_lat, Stream Two: clip_lingunet_lat, Stream Fusion: add
Transport FCN - Stream One: plain_resnet_lat, Stream Two: clip_lingunet_lat, Stream Fusion: conv
Transport FCN - Stream One: plain_resnet_lat, Stream Two: clip_lingunet_lat, Stream Fusion: conv
Traceback (most recent call last):
  File "vlm/scripts/cliport_test_original.py", line 307, in <module>
    dummy = task._task.task_base
AttributeError: 'PickCubeShape' object has no attribute 'task_base'
QMutex: destroying locked mutex

When I run the drawer task, I got this error.

Traceback (most recent call last):
  File "vlm/scripts/cliport_test_original.py", line 308, in <module>
    WriteCustomDataBlock(dummy.get_handle(),"test_a","test_b")
  File "/home/initial/workspace/VLMbench/amsolver/backend/utils.py", line 276, in WriteCustomDataBlock
    pyrep_utils.script_call('_WriteCustomDataBlock@PyRep', PYREP_SCRIPT_TYPE, 
  File "/home/initial/.pyenv/versions/VLMbench/lib/python3.8/site-packages/pyrep/backend/utils.py", line 65, in script_call
    return sim.simExtCallScriptFunction(
  File "/home/initial/.pyenv/versions/VLMbench/lib/python3.8/site-packages/pyrep/backend/sim.py", line 698, in simExtCallScriptFunction
    _check_return(ret)
  File "/home/initial/.pyenv/versions/VLMbench/lib/python3.8/site-packages/pyrep/backend/sim.py", line 27, in _check_return
    raise RuntimeError(
RuntimeError: The call failed on the V-REP side. Return value: -1
QMutex: destroying locked mutex

[KzZheng]

I'm not sure about the first error. But at least the second error indicates that you are not using the correct Coppeliasim and Lua files. Please check the PyRep's install steps to see whether you have finished all the steps, e.g., the path to Coppellasim:

export COPPELIASIM_ROOT=EDIT/ME/PATH/TO/COPPELIASIM/INSTALL/DIR
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$COPPELIASIM_ROOT
export QT_QPA_PLATFORM_PLUGIN_PATH=$COPPELIASIM_ROOT

And simAddOnScript_PyRep.lua should be directly copied to the CoppeliaSim's main folder after PyRep installment.

[KzZheng]

The first error is caused by typos in the code. You can change the test code block to

from amsolver.backend.utils import ReadCustomDataBlock, WriteCustomDataBlock
dummy = task._task.get_base()
WriteCustomDataBlock(dummy.get_handle(),"test_a","test_b")
b = ReadCustomDataBlock(dummy.get_handle(), "test_a")
print(b)

And then you should see the same issue across all tasks.

[daichisaito-cs]

I see, the cause is now clear.
I'll try to fix it later and let you know if it works fine.
Thank you so much.

[KzZheng]

Sure. You can check whether the function _ReadCustomDataBlock is inside the simAddOnScript_PyRep.lua in the correct CoppeliaSim folder to see whether you have the correct lua file. The original PyRep.lua do not have this funciton

[daichisaito-cs]

Hi @KzZheng ,

I tried doing PyRep's install steps again, but it's still not working yet. I put PyRep and CoppeliaSim directories below VLMbench directory, but is that correct?

Thanks

[KzZheng]

I do not put PyRep and CoppeliaSim directories below the VLMbench directory. I downloaded them in two separate folders. PyRep read the environment variable COPPELIASIM_ROOT to open the CoppeliaSim. Can you confirm you can run some examples in the PyRep first?

Also, if you have a monitor, you can set headless=False in env = Environment(action_mode, obs_config=obs_config, headless=True) to visualize the CoppeliaSim. It also includes more error details inside the CoppeliaSim UI.

[daichisaito-cs]

I could run the example in the PyRep with CoppeliaSim visualization.
I also tried putting PyRep and CoppeliaSim directories in different places separately, but nothing has changed.

[daichisaito-cs]

I finally managed to solve the problems!

It appears that simAddOnScript_PyRep.lua file within VLMbench was somehow overwritten with the original PyRep.lua file. So the command "cp ./simAddOnScript_PyRep.lua /Path/To/CoppeliaSim" did not copy the correct content.
I apologize for overlooking such a basic issue.

I'm actually considering using VLMbench for my research on manipulation tasks since I believe it's a highly useful benchmark.

I really appreciate your help.

[KzZheng]

Good! Thank you for your interest in this project!