[transformer.py] online transformation?

[Kerkil]

Hi Xavier,

Does "transformer.py" in your repo perfom the transformations of the dataset ON THE FLY or UP-FRONT? I'd like to perform the data augmentation via various transformations of the original dataset on the fly without needing to store them in GPU: i.e., transform them on CPU and send them to GPU and throw them away after using them up for SGD. If I augment the data in a "preprocessing" fashion upfront by transforming and saving, I would run out of memory for GPU for the data sets I am dealing with.

I am curious if your code does ONLINE transformations of the original data sets that can be used for data augmentation on the fly.

Thanks for your help.

Kerkil

[bouthilx]

There is both of them. I wrote a short post for both here:

Blow up the training set (and my RAM)
Blow up the training set (but NOT my RAM)

Feel free if you have any questions after reading those short posts