Question about training time cost

[DachunKai]

Hello,

Could you please specify the training time cost for the IART model on REDS, such as the IART_REDS_BI_N16.pth? I'm interested in knowing what type of GPU was used, the number of GPUs involved, and approximately how many days the training took.

Thank you!

[PhuTran1005]

Hello,

Could you please specify the training time cost for the IART model on REDS, such as the IART_REDS_BI_N16.pth? I'm interested in knowing what type of GPU was used, the number of GPUs involved, and approximately how many days the training took.

Thank you!

[kai422]

Training the model IART_REDS_BI_N16.pth on a V100 took four weeks. Since I only had 16Gx8 of VRAM available, I enabled gradient checkpointing, which is a memory-saving technique. However, this also means that during backpropagation, the forward pass gets recalculated. If you have more modern GPUs with larger VRAM (so you can disable gradient checkpointing), such as an A100 80G, I estimate that the training could be completed within a week.

Regards,
Kai

[PhuTran1005]

Thank you for your detailed response.