Copy error while training from a remote machine

[a-poduval]

Hi Sri,

We have been running training on Linux. Occasionally, the method copy_json_to_server (it is defined in common/utils.py and is called in environment_randomization/game_config_handler_class) returns an error. The os.system call returns -1 instead of 0 in the midst of training even though write permissions have been granted for the Windows shared directories. To confirm that write permissions were granted, and to try replicating the issue, I executed the copy right after the fail in a python interactive window and it returned 0. Besides, I added prints to check if copies were occurring, and dozens of copies exit successfully before we see this kind of error.

If you recollect, could you let me know as to why do we perform this copy? It seems to be specifically for cases when we have the rl framework and environment generator on separate machines. However, I don't see the path to unreal_host_shared_dir used anywhere except in this file and settings.py; the settings file states that it is for killing and restarting the game, which I believe is currently not supported when airlearning-rl is not run locally. Further, the game config handler class has this copy function called twice in quick succession, do you recollect the reason for this? Could the second one possibly be within the if condition for use_preloaded_json, (which also seems to be called twice, are both needed)?

Btw, just to confirm - the unreal_host_shared_dir path is the shared project folder (basically <AIRLEARNING_ROOT>/airlearning-ue4), right? Or is it supposed to be the UE4 base directory C:/---/UE4.18, or something else altogether?

Thanks,
Ashwin