From 47bcc7297f082e37abb117a3d749a4c940bdb3f9 Mon Sep 17 00:00:00 2001
From: Mingyu Cao <1054527636@qq.com>
Date: Wed, 4 Dec 2024 10:09:56 +0800
Subject: [PATCH] mod: update readme
---
README.md | 30 +++++++++++++++++++++---------
images/figure3.png | Bin 313425 -> 0 bytes
2 files changed, 21 insertions(+), 9 deletions(-)
delete mode 100644 images/figure3.png
diff --git a/README.md b/README.md
index e88c3d346..95f44d46e 100644
--- a/README.md
+++ b/README.md
@@ -11,14 +11,23 @@ Official PyTorch implementation of CD-MOE, as presented in our paper: CONDENSE,
Mixture-of-Experts (MoE) has garnered significant attention for their ability to scale up neural networks while utilizing the same or even fewer active parameters. However, MoE does not relieve the massive memory requirements of networks, which limits their practicality in real-world applications, especially in the era of large language models (LLMs). While recent work explores the possibility of removing entire layers of MoE to reduce memory, the performance degradation is still notable. In this paper, we propose Condense-MoE (CD-MoE) that, instead of dropping the entire MoE layer, condenses the big, sparse MoE layer into a small but dense layer with only a few experts that are activated for all tokens. Our approach is specifically designed for fine-grained MoE with shared experts, where Feed-Forward Networks are split into many small experts, with certain experts isolated to serve as shared experts that are always activated. We demonstrate the effectiveness of our method across multiple MoE models such as DeepSeekMoE and QwenMoE on various benchmarks. Specifically, for the DeepSeekMoE-16B model, our approach maintains nearly 90% of the average accuracy while reducing memory usage by 30% and enhancing inference speed by 30%. Moreover, we show that with lightweight expert fine-tuning, the pruned model can achieve further improvements on specific tasks.
## Results
-
+
+
+
+
+