Experimental reproduction problem

[chen254p]

Hello author, thank you for providing the implementation of your work. While reproducing the results mentioned in the paper, I encountered a few questions:

1. Could you please share details about the training setup? what device was used for training and how many episodes did the training process go through?
2. During testing, I observed that the same model exhibits big performance variation across the different testSet_20A_50T. Would it be possible to provide the complete testSet used in your experiments?

Thank you in advance for your time and support.

[waynedai97]

Hi there,
Thanks for your interest in our work.

1. The model in this repo was trained on GTX 1070Ti+ i7-9700K, the training efficiency is a bit low as we cannot fully use all the resources on GPU currently and the specific model we trained took around 6 days for 2M episodes. (We tried SAC but the results are not as good as REINFORCE)
2. Do you mean the performance variance for each instance or the results for each run? I don't think there is a huge variance in the average for each run, but there is some variance in the performance of each instance, as the difficulty varies. Our testing set follows the following code, basically, we set the seed from 0-49 for the 50 testing instances, you can put this at the bottom of env/task_env.py and you should be able to generate the same set

if __name__ == '__main__':
     import pickle
     import os
     testSet = 'testSet_40A_100T_CONDET'
     if not os.path.exists(f'../{testSet}'):
         os.makedirs(f'../{testSet}')
         for i in range(50):
             env = TaskEnv((40, 40), tasks_range=(100, 100), traits_dim=1, coalition_size=5, seed=i)
             pickle.dump(env, open(f'../{testSet}/env_{i}.pkl', 'wb'))