one-off runs #3
Comments
|
Performance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) |
Performance Dashboard for amp precision (max autotune, with cold start)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
Accuracy warnings Performance speedup warnings Compilation latency (sec) warnings Peak Memory Compression Ratio warnings torchbench suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Build Summarysee moreRun nameday_079_20_03_23_performance_amp_778 Commit hashespytorch commit: 9423b863f800c6d20b9b3de4422558cbb338fb83 TorchDynamo config flagsTorch versiontorch: 2.1.0a0+git9423b86 Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (inference, no max-autotune)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
Accuracy warnings Performance speedup warnings Compilation latency (sec) warnings Peak Memory Compression Ratio warnings torchbench suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_087_28_03_23_performance_amp_838/torchbench_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_087_28_03_23_performance_amp_838/huggingface_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_087_28_03_23_performance_amp_838/timm_models_amp.png : Build Summarysee moreRun nameday_087_28_03_23_performance_amp_838 Commit hashespytorch commit: 0c78456e24eab0e175cec7567d2dfa45ecff58dc TorchDynamo config flagsTorch versiontorch: 2.1.0a0+gita7c8d25 Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (inductor max-autotune with cudagraphs)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
torchbench suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_095_05_04_23_performance_amp_283/torchbench_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_095_05_04_23_performance_amp_283/huggingface_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_095_05_04_23_performance_amp_283/timm_models_amp.png : Build Summarysee moreRun nameday_095_05_04_23_performance_amp_283 Commit hashespytorch commit: f55e72c0f6bd6da016aaa51de379e6ba6d7891cc TorchDynamo config flagsTorch versiontorch: 2.1.0a0+gitf55e72c Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (inductor max-autotune without cudagraphs)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
torchbench suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_095_05_04_23_performance_amp_233/timm_models_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_095_05_04_23_performance_amp_233/huggingface_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_095_05_04_23_performance_amp_233/torchbench_amp.png : Build Summarysee moreRun nameday_095_05_04_23_performance_amp_233 Commit hashespytorch commit: f55e72c0f6bd6da016aaa51de379e6ba6d7891cc TorchDynamo config flagsTorch versiontorch: 2.1.0a0+gitf55e72c Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (inductor max-autotune comparison on timm models)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
Compilation latency (sec) warnings Peak Memory Compression Ratio warnings timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_100_10_04_23_performance_amp_441/timm_models_amp.png : Build Summarysee moreRun nameday_100_10_04_23_performance_amp_441 Commit hashespytorch commit: f55e72c0f6bd6da016aaa51de379e6ba6d7891cc TorchDynamo config flagsTorch versiontorch: 2.1.0a0+gitf55e72c Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (2.0 release binary oneoff)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
Accuracy warnings Performance speedup warnings Compilation latency (sec) warnings Peak Memory Compression Ratio warnings torchbench suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_100_10_04_23_performance_amp_229/huggingface_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_100_10_04_23_performance_amp_229/timm_models_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_100_10_04_23_performance_amp_229/torchbench_amp.png : Build Summarysee moreRun nameday_100_10_04_23_performance_amp_229 Commit hashespytorch commit: f55e72c0f6bd6da016aaa51de379e6ba6d7891cc TorchDynamo config flagsTorch versiontorch: 2.1.0a0+gitf55e72c Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for float32 precision (2.0 release binary oneoff)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
Accuracy warnings Performance speedup warnings Compilation latency (sec) warnings Peak Memory Compression Ratio warnings torchbench suite with float32 precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with float32 precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with float32 precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_100_10_04_23_performance_float32_549/huggingface_float32.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_100_10_04_23_performance_float32_549/torchbench_float32.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_100_10_04_23_performance_float32_549/timm_models_float32.png : Build Summarysee moreRun nameday_100_10_04_23_performance_float32_549 Commit hashespytorch commit: f55e72c0f6bd6da016aaa51de379e6ba6d7891cc TorchDynamo config flagsTorch versiontorch: 2.1.0a0+gitf55e72c Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (inductor max-autotune comparison on timm models, small)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
Compilation latency (sec) warnings Peak Memory Compression Ratio warnings timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_101_11_04_23_performance_amp_691/timm_models_amp.png : Build Summarysee moreRun nameday_101_11_04_23_performance_amp_691 Commit hashespytorch commit: f55e72c0f6bd6da016aaa51de379e6ba6d7891cc TorchDynamo config flagsTorch versiontorch: 2.1.0a0+gitf55e72c Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (inductor max-autotune comparison on timm models, small, ran locally)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
Compilation latency (sec) warnings Peak Memory Compression Ratio warnings timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_103_13_04_23_performance_amp_325/timm_models_amp.png : Build Summarysee moreRun nameday_103_13_04_23_performance_amp_325 Commit hashespytorch commit: 75f55ca63bd5623352c8eda8e31ff76ee5c960a7 TorchDynamo config flagsTorch versiontorch: 2.1.0a0+git75f55ca Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (inductor max-autotune comparison on all suites, with warm start)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Warningssee moreWe flag models where:
Accuracy warnings Performance speedup warnings Compilation latency (sec) warnings Peak Memory Compression Ratio warnings torchbench suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_104_14_04_23_performance_amp_147/torchbench_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_104_14_04_23_performance_amp_147/huggingface_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_104_14_04_23_performance_amp_147/timm_models_amp.png : Build Summarysee moreRun nameday_104_14_04_23_performance_amp_147 Commit hashespytorch commit: 75f55ca63bd5623352c8eda8e31ff76ee5c960a7 TorchDynamo config flagsTorch versiontorch: 2.1.0a0+git75f55ca Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Performance Dashboard for amp precision (Python 3.11)Executive Summarysee moreWe evaluate different backends across three benchmark suites - torchbench, huggingface and timm. We run these experiments on A100 GPUs. Each experiment runs one iteration of forward pass and backward pass for training and forward pass only for inference. For accuracy, we check the numerical correctness of forward pass outputs and gradients by comparing with native pytorch. We measure speedup by normalizing against the performance of native pytorch. We report mean compilation latency numbers and peak memory footprint reduction ratio.Caveats
To measure performance, compilation latency and memory footprint reduction, we remove the models that fail accuracy checks. Passrate Geometric mean speedup Mean compilation time (seconds) Peak memory footprint compression ratio (higher is better) Summary Statistics Diffsee moreFor each relevant compiler, we compare the summary statistics for the most 2 recent reports that actually run the compiler.Current report name: /data/home/williamwen/cluster/oneoff_cron_logs/day_047_16_02_23_performance_amp_184 Previous report name: /data/home/williamwen/cluster/oneoff_cron_logs/day_040_09_02_23_performance_amp_322 Passrate diff Geometric mean speedup diff Warningssee moreWe flag models where:
Accuracy warnings Performance speedup warnings Compilation latency (sec) warnings Peak Memory Compression Ratio warnings Metrics over timesee more/data/home/williamwen/cluster/oneoff_cron_logs/day_144_24_05_23_performance_amp_246/comp_time_over_time.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_144_24_05_23_performance_amp_246/geomean_over_time.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_144_24_05_23_performance_amp_246/passrate_over_time.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_144_24_05_23_performance_amp_246/memory_over_time.png : Recent Regressionssee moreFor each relevant compiler, we compare the most recent 2 reports (that actually run the compiler) to find previously unflagged models that are now flagged as problematic (according to the 'Warnings' section).Regressions for torchbenchCurrent report name (compiler: inductor_no_cudagraphs, suite: torchbench): /data/home/williamwen/cluster/oneoff_cron_logs/day_047_16_02_23_performance_amp_184 Previous report name (compiler: inductor_no_cudagraphs, suite: torchbench): /data/home/williamwen/cluster/oneoff_cron_logs/day_040_09_02_23_performance_amp_322 Current report name (compiler: inductor, suite: torchbench): /data/home/williamwen/cluster/oneoff_cron_logs/day_047_16_02_23_performance_amp_184 Previous report name (compiler: inductor, suite: torchbench): /data/home/williamwen/cluster/oneoff_cron_logs/day_040_09_02_23_performance_amp_322 Accuracy regressions Performance speedup regressions Compilation latency (sec) regressions Peak Memory Compression Ratio regressions Regressions for huggingfaceCurrent report name (compiler: inductor_no_cudagraphs, suite: huggingface): /data/home/williamwen/cluster/oneoff_cron_logs/day_047_16_02_23_performance_amp_184 Previous report name (compiler: inductor_no_cudagraphs, suite: huggingface): /data/home/williamwen/cluster/oneoff_cron_logs/day_040_09_02_23_performance_amp_322 Current report name (compiler: inductor, suite: huggingface): /data/home/williamwen/cluster/oneoff_cron_logs/day_047_16_02_23_performance_amp_184 Previous report name (compiler: inductor, suite: huggingface): /data/home/williamwen/cluster/oneoff_cron_logs/day_040_09_02_23_performance_amp_322 Accuracy regressions Performance speedup regressions Compilation latency (sec) regressions Peak Memory Compression Ratio regressions Regressions for timm_modelsCurrent report name (compiler: inductor_no_cudagraphs, suite: timm_models): /data/home/williamwen/cluster/oneoff_cron_logs/day_047_16_02_23_performance_amp_184 Previous report name (compiler: inductor_no_cudagraphs, suite: timm_models): /data/home/williamwen/cluster/oneoff_cron_logs/day_040_09_02_23_performance_amp_322 Current report name (compiler: inductor, suite: timm_models): /data/home/williamwen/cluster/oneoff_cron_logs/day_047_16_02_23_performance_amp_184 Previous report name (compiler: inductor, suite: timm_models): /data/home/williamwen/cluster/oneoff_cron_logs/day_040_09_02_23_performance_amp_322 Performance speedup regressions Compilation latency (sec) regressions Peak Memory Compression Ratio regressions torchbench suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) huggingface suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) timm_models suite with amp precisionsee morePerformance speedup Accuracy Compilation latency (sec) Peak Memory Compression Ratio Absolute latency (ms) Performance graphssee more/data/home/williamwen/cluster/oneoff_cron_logs/day_144_24_05_23_performance_amp_246/timm_models_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_144_24_05_23_performance_amp_246/torchbench_amp.png : /data/home/williamwen/cluster/oneoff_cron_logs/day_144_24_05_23_performance_amp_246/huggingface_amp.png : Build Summarysee moreRun nameday_144_24_05_23_performance_amp_246 Commit hashespytorch commit: a370bca9a97aaf3c1bc36adbc2d68428fde8e74c TorchDynamo config flagsTorch versiontorch: 2.1.0a0+git956bd03 Environment variablesTORCH_CUDA_ARCH_LIST = 8.0 GPU detailsCUDNN VERSION: 8401 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(next 2 comments are for max-autotune, warm start run)
AMP RUN
Geometric mean speedup
Mean compilation time (seconds)
Peak memory footprint compression ratio (higher is better)
The text was updated successfully, but these errors were encountered: