question about MeZO-adam

[zhaoaustin]

Hi! I find MeZO-adam code in medium size folder, but it uses the Adam from pytorch.optim. Its not like the case in large_models that author re-write the inner_loop. Can you please explain it? Thank you every much for your time and efforts.

[gaotianyu1350]

Hi,

We found in the medium model experiments that Adam is only comparable or worse compared to SGD. Hence we only ran it as an ablation (on the medium-sized models) and did not implement the efficient version.