Video Rendering

[YFCodeDream]

Thanks for your excellent work! I'm new to embodied AI field. Currently I run the starter_code.py successfully, but I wanna render the result to a representation video, just like the More Agent Examples in the project page [https://vis-www.cs.umass.edu/hazard/]. How can I do this?

Looking forward to your reply!

[zhouqqhh]

Hi,
You can try to turn on "record_with_agents" parameter in the submit function, and add "demo" as output directory's suffix . The program will output one image per frame in the output dir. Let me know if you have further questions.

[YFCodeDream]

Thank you for your prompt reply! I'm trying LLM-based agent, please keep this question open for further consultation.

[YFCodeDream]

Hi! I tried to experiment with LLM-v2 agent driven by gpt-3.5-turbo API and render the representation video, so I modified starter_code.py as follows:

from HAZARD import submit

submit(output_dir="llmv2-outputs-demo/",
       env_name="fire",
       agent="llmv2",
       port=1071,
       max_test_episode=1,
       record_with_agents=True)

However, the program terminates after running to Environment Reset. The console output is as follows:

2024-09-20 14:47:48,003 - fire_llmv2_09/20/2024, 14:47:47 - INFO - Environment Created
2024-09-20 14:47:48,003 - fire_llmv2_09/20/2024, 14:47:47 - INFO - done
llm-outputs-demo/mm_craftroom_2a-1
OUTPUT_DIR llm-outputs-demo/mm_craftroom_2a-1
2024-09-20 14:47:48,003 - fire_llmv2_09/20/2024, 14:47:47 - INFO - Episode: 1/1
2024-09-20 14:47:48,003 - fire_llmv2_09/20/2024, 14:47:47 - INFO - Resetting Environment ... data is /HDD-a/gyf/HAZARD/src/HAZARD/data/room_setup_fire/mm_craftroom_2a-1
You need to launch your own build.
Controller connected
object 451306954 not recorded, may be caused by composite objects which you can ignore
object 90574932 not recorded, may be caused by composite objects which you can ignore
object 1298028780 not recorded, may be caused by composite objects which you can ignore
object 969879779 not recorded, may be caused by composite objects which you can ignore
object 1296671630 not recorded, may be caused by composite objects which you can ignore
object 1878811444 not recorded, may be caused by composite objects which you can ignore
object 1145120857 not recorded, may be caused by composite objects which you can ignore
object 703415617 not recorded, may be caused by composite objects which you can ignore
object 60166369 not recorded, may be caused by composite objects which you can ignore
object 10170055 not recorded, may be caused by composite objects which you can ignore
object 1014820023 not recorded, may be caused by composite objects which you can ignore
object 536758888 not recorded, may be caused by composite objects which you can ignore
object 1126821352 not recorded, may be caused by composite objects which you can ignore
object 906371918 not recorded, may be caused by composite objects which you can ignore
object 1755311175 not recorded, may be caused by composite objects which you can ignore
object 384965946 not recorded, may be caused by composite objects which you can ignore
object 351435325 not recorded, may be caused by composite objects which you can ignore
object 781679555 not recorded, may be caused by composite objects which you can ignore
object 998681676 not recorded, may be caused by composite objects which you can ignore
object 747099685 not recorded, may be caused by composite objects which you can ignore
init: {14640280: False, 14544403: False, 16728828: False, 16381463: False, 10511794: False, 10844722: False, 473176: False, 474390: False, 16013534: False, 6999109: False, 6798686: False}
2024-09-20 14:55:06,315 - fire_llmv2_09/20/2024, 14:47:47 - INFO - Environment Reset. Took 438.31182503700256 secs

I checked the code and found that in src/HAZARD/challenge.py Line 280-283:

if "demo" in self.output_dir:
      for i in range(1500):
          self.env.controller.communicate([])
      return

The return keyword may lead to the termination. Therefore, I want to know what the codes in src/HAZARD/challenge.py Line 280-283 do? How can I get the correct actions and video?

Looking forward to your reply!

[EEEEEericKKK]

if "demo" in self.output_dir:
      for i in range(1500):
          self.env.controller.communicate([])
      return

Adding "demo" to the output directory will disable all agent actions and is used to demonstrate environmental changes. If you want to record agents, you can either not add "demo" or manually delete the lines of codes.