Question about Zero supported optimizer

[cccc0der]

Hi, I used came-optimizer to pretrain llm with zero stage 1 and met a untested-warnning given by zero.

It seems only optimizers in support-list are tested to work correctly.

From my understanding, Zero Stage-1 is a wrapper for optimizer to part and gather optimizer status, it should have no relation with specific optimizer type, do I misunderstand it? does it need to be modified for compatibility?

ZERO_SUPPORTED_OPTIMIZERS = [
    torch.optim.Adam, torch.optim.AdamW, FusedAdam, DeepSpeedCPUAdam, torch.optim.Adagrad, DeepSpeedCPUAdagrad
]

        if zero_enabled:
            if not is_zero_supported_optimizer(basic_optimizer):
                assert (
                    self.zero_allow_untested_optimizer()
                ), 'You are using an untested ZeRO Optimizer. Please add <"zero_allow_untested_optimizer": true> in the configuration file to use it.'

                if self.global_rank == 0:
                    logger.warning("**** You are using ZeRO with an untested optimizer, proceed with caution *****")