Results on WSC and WIC datasets cannot be reproduced on OPT-13B with MeZO

[MathIsAll]

Hello,

Thank you for your fantastic work. When I run mezo.sh for WSC and WIC on OPT-13B with MeZO, the reported results in paper cannot be reproduced.

I run mezh.sh on 4 x A100, with per_device_batch = 4, lr = 1e-6, eps = 1e-3.

I want to know more details about the training settings for the reproduction.

Thanks!

[gaotianyu1350]

Hi,

Can you provide more details on your run, for example, what is the result that you got?

BTW, in our experiments, we only used one A100 for OPT-13B.

[gaotianyu1350]

Hi, I just realized that for WSC you should use 1e-7/1e-3. Note that for all OPT-13B MeZO experiments we do a grid search over LR=1e-6/1e-7.

[gaotianyu1350]

FYI, here are the results I got

WSC 1e-6/1e-3

"accuracy": 0.5384615384615384,

WSC 1e-7/1e-3

"accuracy": 0.6346153846153846,

WIC 1e-6/1e-3

"accuracy": 0.6112852664576802,

WIC 1e-7/1e-3

"accuracy": 0.5736677115987461,

[sglucas]

Hi, May I ask for the seed of WSC and WIC. I find the default seed is 0 and I cannot reproduce the result of WIC 61.1.

[gaotianyu1350]

The seed we used for the experiment is 0. Note that different hardwares may lead to slightly different results, even if the random seeds are the same.