model collapse and image blurring

[mapengsen]

I want to generate molecular images based on molecular images, but after I trained 80kimg, the model began to collapse, and the generated images were not clear. How many kimgs did you train at that time and then began to converge?
real image:

generate image :

[nom57]

I had this same issue and its caused either by :
wrong plugin versions , your hardware being unstable, MKL running parallel , or Diffaug being broken in the model
you can pinpoint exactly when the model started to collapse in stats.jsonl

if your scores/fake = NaN that means you have a library/plugin/mkl conflict
if your scores/fake are constantly getting a more and more negative number , that means Augmentation is broken.
(I have a suspicion there is something wrong with augmentation in the model...)

[nom57]

@mapengsen please share the text of your stats.jsonl file.

[mapengsen]

{"Loss/scores/fake": {"num": 1312000, "mean": -4.672473785028225, "std": 2.1516482762262013}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9828719512195122, "std": 0.18428979219139854}, "Loss/G/loss": {"num": 2000, "mean": 18.6724349565506, "std": 3.7821310307624945}, "Loss/scores/real": {"num": 656000, "mean": 5.545181464427855, "std": 2.436094622720647}, "Loss/signs/real": {"num": 656000, "mean": 0.9719756097560975, "std": 0.23508171779035134}, "Loss/D/loss": {"num": 2000, "mean": 0.2549195118771688, "std": 0.5003099562855534}, "Progress/tick": {"num": 1, "mean": 40.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 164.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 1702.18701171875, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1635.7618408203125, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 408.9404602050781, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 66.42518615722656, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.329277038574219, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 9.622943878173828, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 9.755859375, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 0.4728297293186188, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.019701238721609116, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 550.1205444335938, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.001728000002913177, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.69061279296875, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.30079984664917, "std": 0.0}, "timestamp": 1662131910.3168495}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.525232802600395, "std": 2.1465412026797983}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9788307926829268, "std": 0.2046711491532529}, "Loss/G/loss": {"num": 2000, "mean": 18.03233481788635, "std": 3.9396342501069497}, "Loss/scores/real": {"num": 656000, "mean": 5.4938865962493715, "std": 2.525354538533014}, "Loss/signs/real": {"num": 656000, "mean": 0.9669786585365854, "std": 0.25485735997766645}, "Loss/D/loss": {"num": 2000, "mean": 0.3012499381717826, "std": 0.503923548134568}, "Progress/tick": {"num": 1, "mean": 41.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 168.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 7054.953125, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1622.9368896484375, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.7342224121094, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 3729.829345703125, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416057586669922, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 5.309906482696533, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 8.60546875, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 1.9597092866897583, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.08165454864501953, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 546.05224609375, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.0030720001086592674, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.836669921875, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.356095790863037, "std": 0.0}, "timestamp": 1662133533.2627313}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.485393495164266, "std": 2.0739742561584036}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9804817073170732, "std": 0.19661032937411296}, "Loss/G/loss": {"num": 2000, "mean": 17.932043487548828, "std": 3.6148285252373307}, "Loss/scores/real": {"num": 656000, "mean": 5.342033458924875, "std": 2.4559876459530505}, "Loss/signs/real": {"num": 656000, "mean": 0.9680274390243903, "std": 0.25084432880948376}, "Loss/D/loss": {"num": 2000, "mean": 0.2860598468919502, "std": 0.527301423537063}, "Progress/tick": {"num": 1, "mean": 42.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 172.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 8677.8603515625, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1622.898193359375, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.72454833984375, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.009002447128295898, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416103363037109, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853830814361572, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 2.4105167388916016, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.1004381999373436, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 545.159912109375, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.0022720000706613064, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.984375, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.418560028076172, "std": 0.0}, "timestamp": 1662135156.16987}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.63049454630875, "std": 2.1739037362840534}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9784801829268293, "std": 0.20634081423576559}, "Loss/G/loss": {"num": 2000, "mean": 18.441989107370375, "std": 3.886092447679056}, "Loss/scores/real": {"num": 656000, "mean": 5.429873399152989, "std": 2.578254101221881}, "Loss/signs/real": {"num": 656000, "mean": 0.9652682926829268, "std": 0.2612606421583388}, "Loss/D/loss": {"num": 2000, "mean": 0.3172737591400055, "std": 0.5824184042765668}, "Progress/tick": {"num": 1, "mean": 43.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 176.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 10300.291015625, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1622.4207763671875, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.6051940917969, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.009999275207519531, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.4161376953125, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.854516983032227, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 2.861191987991333, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.11921633034944534, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 545.2468872070312, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.0010239999974146485, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.76605224609375, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.331967830657959, "std": 0.0}, "timestamp": 1662136778.6006296}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.481336983247501, "std": 2.1070613859100593}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9790274390243903, "std": 0.20372843109724229}, "Loss/G/loss": {"num": 2000, "mean": 17.869801379680634, "std": 3.5855868879361887}, "Loss/scores/real": {"num": 656000, "mean": 5.456721378698582, "std": 2.487912195247145}, "Loss/signs/real": {"num": 656000, "mean": 0.9655426829268293, "std": 0.2602447452811687}, "Loss/D/loss": {"num": 2000, "mean": 0.3144302575881247, "std": 0.6873845421112359}, "Progress/tick": {"num": 1, "mean": 44.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 180.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 11922.640625, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1622.3408203125, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.585205078125, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.009000062942504883, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416191101074219, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853907108306885, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 3.311844825744629, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.13799352943897247, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 543.815673828125, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.001728000002913177, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.8042907714844, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.405248165130615, "std": 0.0}, "timestamp": 1662138400.9494014}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.682906132930663, "std": 2.1412882135056637}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9824390243902439, "std": 0.18658393112791288}, "Loss/G/loss": {"num": 2000, "mean": 18.69930799818039, "std": 3.9884746346926585}, "Loss/scores/real": {"num": 656000, "mean": 5.580692673229589, "std": 2.5151303912779404}, "Loss/signs/real": {"num": 656000, "mean": 0.9708658536585366, "std": 0.23962365116966441}, "Loss/D/loss": {"num": 2000, "mean": 0.26673546523126423, "std": 0.5133791389623045}, "Progress/tick": {"num": 1, "mean": 45.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 184.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 13545.900390625, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1623.252197265625, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.81304931640625, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.0069997310638427734, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.41619873046875, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853976249694824, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 3.7627501487731934, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.15678125619888306, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 545.6558227539062, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.002208000048995018, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 269.6499328613281, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.244480133056641, "std": 0.0}, "timestamp": 1662140024.2085683}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.65056576809069, "std": 2.1119202363979674}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9841814024390244, "std": 0.17716367317583767}, "Loss/G/loss": {"num": 2000, "mean": 18.558905042290686, "std": 3.831640048596782}, "Loss/scores/real": {"num": 656000, "mean": 5.637479898929596, "std": 2.4203280085249137}, "Loss/signs/real": {"num": 656000, "mean": 0.9736463414634147, "std": 0.22806315299694457}, "Loss/D/loss": {"num": 2000, "mean": 0.24197689676519257, "std": 0.5268206440067535}, "Progress/tick": {"num": 1, "mean": 46.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 188.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 15170.62890625, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1624.719482421875, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 406.17987060546875, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.008999824523925781, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416221618652344, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 4.21406364440918, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.17558598518371582, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 549.0623779296875, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.001728000002913177, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 258.6992492675781, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.421023845672607, "std": 0.0}, "timestamp": 1662141648.9370267}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.8030146484375, "std": 2.2973162095130024}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9799557926829269, "std": 0.1992150706828587}, "Loss/G/loss": {"num": 2000, "mean": 19.069496123552323, "std": 4.151811259193795}, "Loss/scores/real": {"num": 656000, "mean": 5.462811172973819, "std": 2.4533227776380713}, "Loss/signs/real": {"num": 656000, "mean": 0.9671798780487805, "std": 0.25409266714635065}, "Loss/D/loss": {"num": 2000, "mean": 0.30352093021915266, "std": 0.5898877808623719}, "Progress/tick": {"num": 1, "mean": 47.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 192.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 16794.46484375, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1623.827392578125, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.95684814453125, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.009032487869262695, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416400909423828, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 4.6651291847229, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.19438038766384125, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 545.2503051757812, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.002047999994829297, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 258.20159912109375, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.412415981292725, "std": 0.0}, "timestamp": 1662143272.7743967}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.6266801119781125, "std": 2.172609992229302}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9808384146341463, "std": 0.19482300782498566}, "Loss/G/loss": {"num": 2000, "mean": 18.464759120941164, "std": 4.052700887234424}, "Loss/scores/real": {"num": 656000, "mean": 5.428200291354481, "std": 2.4608727064049054}, "Loss/signs/real": {"num": 656000, "mean": 0.9674146341463414, "std": 0.25319740448807987}, "Loss/D/loss": {"num": 2000, "mean": 0.2917294027998614, "std": 0.5912144239335322}, "Progress/tick": {"num": 1, "mean": 48.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 196.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 18417.859375, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1623.385498046875, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.84637451171875, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.008039474487304688, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416408538818359, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 5.116071701049805, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.213169664144516, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 545.5564575195312, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.0016959999920800328, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.69677734375, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.4113922119140625, "std": 0.0}, "timestamp": 1662144896.1671762}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.453708056007944, "std": 2.1097174944513855}, "Loss/signs/fake": {"num": 1312000, "mean": -0.979954268292683, "std": 0.19922256914052772}, "Loss/G/loss": {"num": 2000, "mean": 17.685931020736696, "std": 3.5293279692824355}, "Loss/scores/real": {"num": 656000, "mean": 5.331574049647261, "std": 2.4418424818591626}, "Loss/signs/real": {"num": 656000, "mean": 0.9678170731707317, "std": 0.2516547493675383}, "Loss/D/loss": {"num": 2000, "mean": 0.2982898719501663, "std": 0.5327396493262248}, "Progress/tick": {"num": 1, "mean": 49.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 200.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 20041.53125, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1623.66650390625, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.9166259765625, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.006997585296630859, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416408538818359, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 5.567092418670654, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.2319621741771698, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 548.5598754882812, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.026784000918269157, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.0784912109375, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.280640125274658, "std": 0.0}, "timestamp": 1662146519.841446}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.426826917508754, "std": 2.1134708184396}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9776051829268293, "std": 0.210447395599472}, "Loss/G/loss": {"num": 2000, "mean": 17.515529624700545, "std": 3.5166449654954577}, "Loss/scores/real": {"num": 656000, "mean": 5.357669995709164, "std": 2.5384493831926926}, "Loss/signs/real": {"num": 656000, "mean": 0.9620670731707317, "std": 0.27281302520353007}, "Loss/D/loss": {"num": 2000, "mean": 0.34500177393518244, "std": 0.6018819531186338}, "Progress/tick": {"num": 1, "mean": 50.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 204.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 21665.56640625, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1624.0262451171875, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 406.0065612792969, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.008002519607543945, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416408538818359, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 6.018212795257568, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.2507588565349579, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 546.2998046875, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.0009599999757483602, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.8478698730469, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.571135997772217, "std": 0.0}, "timestamp": 1662151866.4775906}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.810085728063816, "std": 2.203256900139646}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9843262195121951, "std": 0.17635728955966015}, "Loss/G/loss": {"num": 2000, "mean": 19.128277977228166, "std": 3.9526178219305743}, "Loss/scores/real": {"num": 656000, "mean": 5.957022389923654, "std": 2.5848501898710454}, "Loss/signs/real": {"num": 656000, "mean": 0.9738902439024391, "std": 0.2270193666444511}, "Loss/D/loss": {"num": 2000, "mean": 0.2422545251833999, "std": 0.5688795859533273}, "Progress/tick": {"num": 1, "mean": 51.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 208.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 27011.509765625, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1623.3331298828125, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.8332824707031, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 3722.60986328125, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.41693115234375, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 7.503197193145752, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.312633216381073, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 548.8342895507812, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.002240000059828162, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.6845397949219, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.212736129760742, "std": 0.0}, "timestamp": 1662153489.8187177}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.837799763191037, "std": 2.1695528014709073}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9862469512195122, "std": 0.165278405153901}, "Loss/G/loss": {"num": 2000, "mean": 19.332257330060006, "std": 3.889203780098971}, "Loss/scores/real": {"num": 656000, "mean": 5.884985626924329, "std": 2.515746167542029}, "Loss/signs/real": {"num": 656000, "mean": 0.9757408536585366, "std": 0.2189287247067189}, "Loss/D/loss": {"num": 2000, "mean": 0.21973969318869876, "std": 0.55292004861535}, "Progress/tick": {"num": 1, "mean": 52.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 212.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 28634.14453125, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1622.627197265625, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.65679931640625, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.008999347686767578, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416938781738281, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 7.953929424285889, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.33141371607780457, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 548.6162109375, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.0030720001086592674, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 276.7219543457031, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.619359970092773, "std": 0.0}, "timestamp": 1662155112.453897}
{"Loss/scores/fake": {"num": 1312000, "mean": -5.082049492510354, "std": 2.332657388817821}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9840548780487804, "std": 0.17786510896856586}, "Loss/G/loss": {"num": 2000, "mean": 20.149699022769926, "std": 4.22588006748906}, "Loss/scores/real": {"num": 656000, "mean": 6.03276230230564, "std": 2.609316319495893}, "Loss/signs/real": {"num": 656000, "mean": 0.9752012195121951, "std": 0.22132008824760455}, "Loss/D/loss": {"num": 2000, "mean": 0.23324685303619117, "std": 0.4873265501644684}, "Progress/tick": {"num": 1, "mean": 53.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 216.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 30257.1484375, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1622.9930419921875, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.7482604980469, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.008999109268188477, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416938781738281, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 8.404763221740723, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.3501984775066376, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 545.6483154296875, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.025599999353289604, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 266.1561584472656, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.40115213394165, "std": 0.0}, "timestamp": 1662156735.455916}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.70151941271526, "std": 2.306700980424207}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9781676829268293, "std": 0.20781718908107158}, "Loss/G/loss": {"num": 2000, "mean": 18.634065569400786, "std": 3.8074436791638226}, "Loss/scores/real": {"num": 656000, "mean": 5.513901919702204, "std": 2.5747868749597864}, "Loss/signs/real": {"num": 656000, "mean": 0.9639603658536585, "std": 0.2660458852590674}, "Loss/D/loss": {"num": 2000, "mean": 0.32726542445600715, "std": 0.6332079610255531}, "Progress/tick": {"num": 1, "mean": 54.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 220.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 31879.0703125, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1621.9129638671875, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.4782409667969, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.009000539779663086, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416938781738281, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 8.855297088623047, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.36897072196006775, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 547.0747680664062, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.015359999611973763, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 256.8243103027344, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.444416046142578, "std": 0.0}, "timestamp": 1662158357.3778982}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.513453964884688, "std": 2.132558722497116}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9796935975609756, "std": 0.2005005109670126}, "Loss/G/loss": {"num": 2000, "mean": 17.919744742631913, "std": 3.792051844889126}, "Loss/scores/real": {"num": 656000, "mean": 5.5132986349478, "std": 2.605148070882686}, "Loss/signs/real": {"num": 656000, "mean": 0.9679176829268292, "std": 0.251267505017179}, "Loss/D/loss": {"num": 2000, "mean": 0.3047439832608125, "std": 0.5494016228898124}, "Progress/tick": {"num": 1, "mean": 55.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 224.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 33502.59765625, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1623.5201416015625, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.8800354003906, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.0070307254791259766, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416954040527344, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 9.30627727508545, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.38776153326034546, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 544.1337890625, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.0023040000814944506, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.3966979980469, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 7.741856098175049, "std": 0.0}, "timestamp": 1662159980.906039}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.475256897763508, "std": 2.0586601898843315}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9801996951219513, "std": 0.19801150896559983}, "Loss/G/loss": {"num": 2000, "mean": 17.753160313367843, "std": 3.5283565860339956}, "Loss/scores/real": {"num": 656000, "mean": 5.441332107119444, "std": 2.515824186392179}, "Loss/signs/real": {"num": 656000, "mean": 0.9694481707317073, "std": 0.24529623777169196}, "Loss/D/loss": {"num": 2000, "mean": 0.28239527040261236, "std": 0.49429597531502756}, "Progress/tick": {"num": 1, "mean": 56.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 228.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 35125.57421875, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1622.967529296875, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.74188232421875, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.008999824523925781, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416954040527344, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 9.75710391998291, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.4065459966659546, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 555.6377563476562, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.002047999994829297, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 258.12890625, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.357120037078857, "std": 0.0}, "timestamp": 1662161603.8815465}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.76585630272656, "std": 2.1491129520345997}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9832804878048781, "std": 0.18209745276142938}, "Loss/G/loss": {"num": 2000, "mean": 18.98432433128357, "std": 3.691129745477171}, "Loss/scores/real": {"num": 656000, "mean": 5.659986590362177, "std": 2.5255073902824576}, "Loss/signs/real": {"num": 656000, "mean": 0.9741189024390244, "std": 0.22603620044362444}, "Loss/D/loss": {"num": 2000, "mean": 0.24980433843084757, "std": 0.5046223783717178}, "Progress/tick": {"num": 1, "mean": 57.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 232.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 36748.2734375, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1622.691650390625, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.67291259765625, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.009001731872558594, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416980743408203, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 10.207854270935059, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.42532724142074585, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 545.6410522460938, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.0017600000137463212, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 258.21185302734375, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.333951950073242, "std": 0.0}, "timestamp": 1662163226.582198}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.468171984027071, "std": 2.081827094009}, "Loss/signs/fake": {"num": 1312000, "mean": -0.978907012195122, "std": 0.2043062932834407}, "Loss/G/loss": {"num": 2000, "mean": 17.732324146270752, "std": 3.7391471531600047}, "Loss/scores/real": {"num": 656000, "mean": 5.586327120856541, "std": 2.5475142093773107}, "Loss/signs/real": {"num": 656000, "mean": 0.9685945121951219, "std": 0.24864567349039843}, "Loss/D/loss": {"num": 2000, "mean": 0.29675778870257274, "std": 0.593137330322738}, "Progress/tick": {"num": 1, "mean": 58.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 236.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 38371.4296875, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1623.1485595703125, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 405.7871398925781, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.009001016616821289, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416980743408203, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 10.658730506896973, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.4441137909889221, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 544.2577514648438, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.002047999994829297, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 257.4637145996094, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 4.423007965087891, "std": 0.0}, "timestamp": 1662164849.740742}
{"Loss/scores/fake": {"num": 1312000, "mean": -4.5586029662155525, "std": 2.08530492261411}, "Loss/signs/fake": {"num": 1312000, "mean": -0.9813003048780488, "std": 0.19248301651379165}, "Loss/G/loss": {"num": 2000, "mean": 18.093166348457338, "std": 3.6233503199248127}, "Loss/scores/real": {"num": 656000, "mean": 5.469760396399149, "std": 2.475573982295461}, "Loss/signs/real": {"num": 656000, "mean": 0.9711859756097561, "std": 0.23832289184827854}, "Loss/D/loss": {"num": 2000, "mean": 0.2641012859315415, "std": 0.4668572424850996}, "Progress/tick": {"num": 1, "mean": 59.0, "std": 0.0}, "Progress/kimg": {"num": 1, "mean": 240.0019989013672, "std": 0.0}, "Timing/total_sec": {"num": 1, "mean": 40014.59375, "std": 0.0}, "Timing/sec_per_tick": {"num": 1, "mean": 1643.1527099609375, "std": 0.0}, "Timing/sec_per_kimg": {"num": 1, "mean": 410.7881774902344, "std": 0.0}, "Timing/maintenance_sec": {"num": 1, "mean": 0.009979963302612305, "std": 0.0}, "Resources/cpu_mem_gb": {"num": 1, "mean": 5.416984558105469, "std": 0.0}, "Resources/peak_gpu_mem_gb": {"num": 1, "mean": 4.853508472442627, "std": 0.0}, "Resources/peak_gpu_mem_reserved_gb": {"num": 1, "mean": 6.3203125, "std": 0.0}, "Progress/augment": {"num": 1, "mean": 0.0, "std": 0.0}, "Timing/total_hours": {"num": 1, "mean": 11.115164756774902, "std": 0.0}, "Timing/total_days": {"num": 1, "mean": 0.4631318747997284, "std": 0.0}, "Timing/Gmain": {"num": 1, "mean": 549.2937622070312, "std": 0.0}, "Timing/Greg": {"num": 1, "mean": 0.001567999948747456, "std": 0.0}, "Timing/Dmain": {"num": 1, "mean": 260.29364013671875, "std": 0.0}, "Timing/Dreg": {"num": 1, "mean": 8.859647750854492, "std": 0.0}, "timestamp": 1662166492.9023833}