Wandb 'steps'

[sasank-desaraju]

Adjust Wandb logging frequency so that it outputs stats in the frequency and organization we want. For example, I believe Wandb currently every batch without doing a good job of separating epochs. We definitely want to calculate validation loss over epochs, not batches. For the validation images, we want them separated by epoch even if we output every image in an epoch.

The solution to this likely includes using the "commit" bool in log(). I.e. wandb_run.log(data={'validation/loss': val_loss}, commit=False). I think this can allow us to control the "step" at which data is pushed. Additionally, perhaps we can output to different keys such as wandb_run.log({'validation/epoch_} + pl_module.current_epoch() + '/loss': val_loss).