Ulysses parallelism

brunomaga · Sep 18, 2024 · d3c0429 · d3c0429
1 parent 0496011
commit d3c0429
Show file tree

Hide file tree

Showing 6 changed files with 314 additions and 365 deletions.
diff --git a/...24-07-11-GPT-lite-sequence-parallelism.md → ...24-08-30-GPT-lite-sequence-parallelism.md b/...24-07-11-GPT-lite-sequence-parallelism.md → ...24-08-30-GPT-lite-sequence-parallelism.md
diff --git a/_posts/2023-08-18-GPT-lite-DeepSpeed-sharding.md b/_posts/2023-08-18-GPT-lite-DeepSpeed-sharding.md
@@ -1,6 +1,6 @@
 ---
 layout: post
-title:  "Distributed training of a GPT model: sharding, offloading, activation checkpointing, and communication quantization via DeepSpeed"
+title:  "Distributed GPT model: sharding, offloading, activation checkpointing, and communication quantization via DeepSpeed"
 categories: [machine learning, Transformer, GPT, DeepSpeed]
 tags: [machinelearning]
 ---

diff --git a/_posts/2023-08-30-GPT-lite-DeepSpeed-pipeline.md b/_posts/2023-08-30-GPT-lite-DeepSpeed-pipeline.md
@@ -1,6 +1,6 @@
 ---
 layout: post
-title:  "Distributed training of a GPT model (part 2): pipeline parallelism via DeepSpeed"
+title:  "Distributed GPT model (part 2): pipeline parallelism via DeepSpeed"
 categories: [machine learning, Transformer, GPT, DeepSpeed]
 tags: [machinelearning]
 ---

diff --git a/_posts/2023-09-02-GPT-lite-Megatron-LM-model-parallelism.md b/_posts/2023-09-02-GPT-lite-Megatron-LM-model-parallelism.md
@@ -1,6 +1,6 @@
 ---
 layout: post
-title:  "Distributed training of a GPT model (part 3): Megatron-LM model parallelism from scratch"
+title:  "Distributed GPT model (part 3): Megatron-LM model parallelism"
 categories: [machine learning, Transformer, GPT, DeepSpeed]
 tags: [machinelearning]
 ---

diff --git a/assets/GPT-lite-distributed/ring_attention_qkv.png b/assets/GPT-lite-distributed/ring_attention_qkv.png