From cc8b52a13bb4153172a429be9d4ed1f034777e3f Mon Sep 17 00:00:00 2001
From: Namrata Shivagunde
 <51484711+NamrataRShivagunde@users.noreply.github.com>
Date: Wed, 8 May 2024 10:19:49 -0400
Subject: [PATCH] Update index.html

---
 docs/2024/pept_relora_n_galore/index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/2024/pept_relora_n_galore/index.html b/docs/2024/pept_relora_n_galore/index.html
index d846244..80cc396 100644
--- a/docs/2024/pept_relora_n_galore/index.html
+++ b/docs/2024/pept_relora_n_galore/index.html
@@ -154,7 +154,7 @@ <h2 id="intro">Parameter Efficient Pre-training (PEPT)</h2>
 
 <h2 id="relora">ReLoRA: High-Rank Training Through Low-Rank Updates</h2>
 
-	<p>ReLoRA uses LoRA (Hu et al., 2022) decomposition technique where the pre-trained model weights are frozen and trainable rank decomposition matrices (W<sub>A</sub>, W<sub>B</sub>) are injected into each layer of the LLM. However in LoRA, the rank of the matrix is restricted by the rank r (given below), and the new trainable parameters (W<sub>A</sub> and W<sub>B</sub>) are merged back to the original matrices only after the end of the training. This suggests the potential to use PEPT techniques to pre-train LLMS. </p>
+	<p>ReLoRA uses LoRA (Hu et al., 2022) decomposition technique where the pre-trained model weights are frozen and trainable rank decomposition matrices (W<sub>A</sub>, W<sub>B</sub>) are injected into each attention and MLP layer of the LLM. However in LoRA, the rank of the matrix is restricted by the rank r (given below), and the new trainable parameters (W<sub>A</sub> and W<sub>B</sub>) are merged back to the original matrices only after the end of the training. </p>
 	
 	<figure>
 		<img src="/blog/assets/images/relora-rank-property.png" />