Skip to content

Commit

Permalink
post: update transformers-vram-comsumption
Browse files Browse the repository at this point in the history
  • Loading branch information
p208p2002 committed Oct 26, 2024
1 parent c61abbc commit f83f921
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion public/docs/transformers-vram-comsumption/document.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,13 +124,16 @@ $$


### Optimizer

優化模型有許多的 optimizer 可以選擇。通常,每個參數需要 4 Bytes 的儲存空間,而某些 optimizer 會儲存額外的資訊。

| Optimizer | First Moments | Second Moments | Bytes per Param |
|------------------------|---------------|----------------|-----------------|
| SGD ||| 4 |
| SGD w momentum ||| 8 |
| ADAM ||| 12 |

我們以常用的 Adam optimizer 計算記憶體占用:
下面以常用的 Adam optimizer 計算記憶體占用:

$$
\text{Optimizer Memory} = \text{Model Params} \times 12\ \text{Bytes} \tag{4}
Expand Down

0 comments on commit f83f921

Please sign in to comment.