MyoChallenge/MyoSuite Q&A

[andreh1111]

Hey challenger!

Welcome to the official discussion thread for MyoChallenge '24. This year's challenge brings bionic models into the spotlight with both manipulation and locomotion tracks, requiring you to coordinate biological and prosthetic limbs. If you have any questions or need clarification about the challenge or the use of MyoSuite, this is the place to find answers. You’re also encouraged to use this space to team up with others to tackle the challenge together if you’re looking for collaborations!

Feel free to ask and interact with the community, be respectful of other participants, and good luck with the challenge! 🦾

[Ruthrash]

Hello MyoChallenge team. I had a couple of clarifications questions on the manipulation track-

• The handover task can be between both hands correct? i.e from myohand to MPL and MPL to myohand. This was not explicitly mentioned anywhere.

• In the observation space listed here, I noticed myo hand's palm and fingertip positions are given whereas for the MPL only the palm's location is given. Shouldn't the MPL's fingertip also be provided?

Please share your thoughts on this.

[cherylwang20]

Hi Ruthrash,

Thanks for posting these questions.

For Q1: The object has to be passed from the myohand to the MPL (the left collar is the starting position with the right collar being the end goal).

For Q2: The myohand's palm and fingertip positions are given as a reference and a guide for you to extract the corresponding MPL attributes on your own based on your needs. Those given attributes are also used for training the baseline, which you can find at the bottom of the tutorial here: https://colab.research.google.com/drive/1AqC1Y7NkRnb2R1MgjT3n4u02EmSPem88#scrollTo=-mAnRvYjIS4d

Hope this helps clarify the question. Thanks!

[Ruthrash]

Thanks a lot for the answers!

[NuCapybara]

Hi MyoChallenge Team,

May I ask when will the registration open？whats the maximum number of teammates can we have? Thanks!

[cherylwang20]

Hi Stella,
Thank you for the question. We plan to start the submission on EvalAI in early September, please stay tuned for that!

In terms of team size, we should be following the rules from past years, that is:
Team formation / Eligibility: There is no limit to the team size and the affiliation: participants from both academic and industrial institutions are welcome. Each participant will be allowed to join only one team during registration and this cannot be changed during the competition.

[NuCapybara]

Thank you so much Huiyi!

[han-li-jun]

Hello MyoChallenge Team,

I have a question on the manipulation track:
In the bimanual_v0.py file, which defines the simulation environment, the 'obj_size_range', 'obj_mass_range', 'obj_friction_range' are set None, I would like to know whether we need to set the range of parameters by ourselves? Or will the official team give a range of changes? How are these parameterizations randomized to test the effectiveness of the algorithm after the final submission of the code?

Please share your thoughts on this.

[cherylwang20]

Hi,

Thank you for your questions. In our current simulation environment, we have intentionally not varied certain parameters to simplify the task.

However, we do plan to introduce variations in these parameters when submissions open officially on EvalAI. We expect that the policy you've trained will generalize well under these new conditions. Please note that for the final submission, we will use a different set of hidden parameters that are not accessible during training.

We encourage you to explore these variations before the official submissions begin. Doing so will not only deepen your understanding but also help us monitor the community's progress and adjust the challenge's difficulty accordingly. Please feel free to share your insights with us if you decide to experiment.

I hope this addresses your concerns.

[han-li-jun]

Thank you very much!

[Shaswat2001]

Hi MyoChallenge Team,

In the MyoChallenge 2024 paper published -
https://openreview.net/pdf?id=2vkf3Wkw2v

For imitation learning. can you provide links to several publicly available datasets that we can use for training.

Regards

[AndresChS]

Hi MyoChallenge team, I had one question about the models.

Reviwing the “env_assets.py”, motor joint are defined by two parameters, ‘gear’ and ‘ctrlrange’. My question is about what implies in real world. What is the real torque generated by the motor following torque-control equation?
Real_torque = gear * ctrl € [ctrlrange]

Thanks for your great job,
Best regards!

[cherylwang20]

@elladyr @P-Schumacher

[elladyr]

@AndresChS Apologies, I must have missed this notification. Yes, you are correct that the torque is given as ctrl * gear

[AndresChS]

Thanks you so much!!!

[sbs0323]

Hello, myosuite team.
we are preparing challenge 24 [Locomotion track].

There is difference between
related document [https://openreview.net/forum?id=2vkf3Wkw2v] and
given environment [https://github.com/MyoHub/myosuite/blob/main/myosuite/envs/myo/myochallenge/run_track_v0.py].
In given python environment file, there is only straight track, without semicircle track

Also, when eval AI submission site will be open?

[elladyr]

@sbs0323 Apologies for the late reply.
Yes, there is a change in the track to simplify the distance tracking metrics. In the current state, the track is a 100m long straight track with a 20m initial flat terrain, and a 80m long track with randomized terrain.

[Mehshida]

Is someone willling to collaborate together?

[ttktjmt]

[for people seeking for a team to join]
@Mehshida, my friend, and I are now collaborating and we are still open to new members! Please reply if you're interested.

[Ruthrash]

I am working on the manipulation track and I'm also looking for teammates. Please feel free to email me at- [email protected] if you're interested :D

[han-li-jun]

Hi, i have a question about the maxmium step in one episode. Is the maximum stride length 250 steps? Can we modify this step length to a larger number ourselves? Thanks a lot.

[cherylwang20]

Hi,

I presume that you are referring to the manipulation track?

Starting from MyoSuite >= 2.8.0, the manipulation track has increased the maximum steps in one episode to 300, which we think is sufficient to complete the task. Nevertheless, feel free to modify this based on your needs during training.

The final evaluation will be on 300 steps unless it is shown that ALL teams cannot complete the task within this timeframe

Best,
Cheryl

[han-li-jun]

Get it. Thank you very much!

[Ruthrash]

Hey,

For the manipulation track, are we allowed to have different policies to control each of the myohand and prosthetic limb (i.e. two policies in total) or are we only allowed to submit one policy?

Best,
Ruthrash

[Balint-H]

Hello, we are adding the disclaimer below to the docs to clarify this. In short we decided not to restrict it to two policies, as we deemed the challenge hard enough already, and wanted people to have as much freedom as possible in designing the policy.

However, you can actually have two policies under the hood in your solution, with a bit of preprocessing that splits up observations and merges actions!It will likely be more challenging, so if you are just starting out I suggest considering using single agent policies first.

Disclaimer on challenge realism:
A single policy is used to map observations about the entire bimanual task to the control of both arms. This differs from real prosthetic control tasks, where the two separate agents (human and device) communicate through a narrow bandwidth human-machine interface (HMI). Usually only 1-4 degrees of freedom are communicated between the two systems, often thresholded into binary channels.

Since bimanual manipulation is already a complex and challenging task, we did not place such a restriction into the environment or submission rules. Single agent solutions should indicate an upper limit to performance with a perfect HMI. We encourage you to investigate implementing more realistic multi-agent scenarios after the challenge!

[cherylwang20]

#NeurIPS #MyoChallenge 2024 Submissions are LIVE!!!

MyoChallengers, you can now register your team on EvalAI, submit your best solution and start competing against teams from everywhere in the world 🌎 Follow the link here to start the Challenge: https://lnkd.in/dFpURga7 !

1. Register a team on EvalAI & participate in MyoChallenge 2024! https://eval.ai/web/challenges/challenge-page/2373/overview
2. Clone our public template for submissions: https://github.com/MyoHub/myochallenge_2024eval
3. Checkout the instructions and submit a testing solution by following the Github/DIY submission.

You can also ask for Google Cloud Credits for computing, just send an email to [email protected] with your Team name.

[Ruthrash]

Hello Myochallenge team,

I had another question related to the manipulation track- referring to object variability here, my understanding is that/ the type of object (jelly box) is fixed but its geometric and physical attributes are varied while evaluating for the competition. There is no variability in the type of object i.e. it is always the jelly box but with different dimensions and physical attributes, is my understanding correct?

thanks in advance,
Ruthrash

[cherylwang20]

Hi Ruthrash,

Yes, the evaluation env only has one type of object (i.e., jelly box). We will only consider additional objects if all teams perform extremely well on all metrics and require an additional tiebreaker to determine the winner.

-Cheryl

[Ruthrash]

thank for the clarification @cherylwang20

[patebel]

Hi @cherylwang20, so did I get it right that during the evaluation stage the size and weight of the jelly box is randomized over multiple trails? Are the range known in which size and weight will be?

Thanks for this awesome challenge!
Patrick

[cherylwang20]

Hi Patrick,

Thanks for the question. If you load the current env, the size & weight of the jelly box are already randomized according to the documentation here: https://myosuite.readthedocs.io/en/latest/challenge-doc.html#observation-space

During evaluation (i.e., when you submit to evalai), there will be another hidden set of randomization parameters in which you won't have access.

Hope this answers your question!

-Cheryl

[sbs0323]

Dear myochallenge team
I have two question.
for locomotion track, there are some initial parameters,

1. below difficulty level vectors are fixed in given file.

'hills_difficulties': (0.0, 0.0, 0.0, 0.0, 0.0, 0.03, 0.0, 0.06, 0.0, 0.09, 0.0, 0.12, 0.0, 0.15, 0.0, 0.18, 0.0, 0.21, 0.0, 0.24, 0.0, 0.0, 0.0, 0.0),
'rough_difficulties': (0.0, 0.0, 0.0, 0.0, 0.0, 0.03, 0.0, 0.06, 0.0, 0.09, 0.0, 0.12, 0.0, 0.15, 0.0, 0.18, 0.0, 0.21, 0.0, 0.24, 0.0, 0.0, 0.0, 0.0),
'stairs_difficulties': (0.0, 0.0, 0.0, 0.0, 0.0, 0.03, 0.0, 0.06, 0.0, 0.09, 0.0, 0.12, 0.0, 0.15, 0.0, 0.18, 0.0, 0.21, 0.0, 0.24, 0.0, 0.0, 0.0, 0.0),

Acoording to eval.ai documentation, variation is only about terrain type.

"Obstacle type are randomized for each episode."

are these three vectors always same for the evaluation environment?

2. reset type has three choices.
   self.reset_type = 'random', 'init', 'osl_init'

'random' is starting from one among 5 key_pos in model file.
'init' is starting from stand pose.
'osl_init' is starting from sample gait cycle.

which reset type would be used in evaluation?

[elladyr]

Hi @sbs0323, thank you for your questions

1. No, the difficulty values given here will be different. Do note that the evaluation will also feature mixed terrains in the randomization. Similar to the past year, the actual difficulty values for evaluation are hidden to encourage the development of generalized policies. These hidden values will be released as a version update after the challenge has ended
2. The reset type for evaluation will be random, which picks from the 3 starting poses in the XML file:
   a. A neutral pose with both legs on the ground
   b. Biological leg at heel strike (biological leg as the front leg)
   c. OSL at heel strike (Prosthetic as the front leg)

[cherylwang20]

🚀 MyoChallenge '24 Leaderboard is now LIVE! 🚀

Curious about the new Phase 2 variations? Need guidance on EvalAI submissions? Got burning questions for the organizers?
Join us at the first MyoChallenge '24 Workshop & Q&A on Sep 30th, 2024 at 10:00 AM EST.

Register now 👇
https://forms.gle/tLHJqsNGWpJ9sNCx6

[cherylwang20]

!ATTENTION MyoChallenge Participants!
🚀 Announcement: MyoChallenge 2024 - MyoSuite v2.8.2 Bugfix Release for Locomotion Track.

🔧 New in v2.8.2: Critical fixes for locomotion observations, including joint angles and heightfields. To Ensure top performance: Update to the latest MyoSuite version before submitting your MyoChallenge entries.

[AndresChS]

Hello MyoChallenge Team!!
I had an issue with GitHub actions. I cloned the repo "myoChallenge_2024eval" and when I tried to run the provided workflows actions I don't find the .yml files so I need to make it. It is normal or I failed following some step?

Thanks!

[elladyr]

Hi @AndresChS could you tell me which action you tried to run and attach a screenshot of the error message? The .yml files for GH actions comes together with the template repository and you should not be creating any files for it

[nu-jliu]

Hi, I have explore the official repo, and I have seen this block of code, just wondering what this block of code doing, I am really confused. We are trying to get the docker running. and we have setup the mujoco simulation using this environment provided.

class EnvironmentServicer(object):
    """Missing associated documentation comment in .proto file."""

    def reset(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

    def get_action_space(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

    def get_observation_space(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

    def get_obsdict(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

    def act_on_environment(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

    def get_output_keys(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

    def set_output_keys(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

    def set_environment_keys(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

    def change_osl_mode(self, request, context):
        """Missing associated documentation comment in .proto file."""
        context.set_code(grpc.StatusCode.UNIMPLEMENTED)
        context.set_details('Method not implemented!')
        raise NotImplementedError('Method not implemented!')

[elladyr]

Hi @nu-jliu,

This block of code is not meant to be used directly as they are the underlying code for the Docker agent to communicate with the Evaluation docker environment.

When submitting your solutions, you are just required to load your policies in the LocoOSL or the ManiMPL file.

[gparc]

Hello Myosuite Team,
Yesterday we tried to submit our agent for evaluation on eval.ai twice, but both failed.
The error log indicates that approximately 20 trials were done before a sudden error occurred.
Since the error was from the evaluation system, we couldn't identify how to resolve it.
Could you give any suggestions or guidance to help us successfully submit our agent?
Thank you.

File "//agent_locoOSL_random.py", line 216, in
base = rc.act_on_environment(action)
File "/utils.py", line 79, in act_on_environment
s = self.stub.act_on_environment(p)
File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 946, in call
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "{"created":"@1728893501.006217117","description":"Error received from peer ipv6:[::1]:8086","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Socket closed","grpc_status":14}"

[elladyr]

Hi @gparc,

Thank you for your submission. Let me check on the evaluation system before getting back to you.
We can troubleshoot this offline one the Slack channel.

[cherylwang20]

!ATTENTION MyoChallenge Participants!
🚀 Announcement: MyoChallenge 2024 - MyoSuite v2.8.3 Bugfix Release for Manipulation Track.
🔧 New in v2.8.3: Critical fixes for evaluation threshold for solved condition.

To Ensure top performance: Update to the latest MyoSuite version before submitting your MyoChallenge entries.