Update index.html

docs/2024/pept_relora_n_galore/index.html

@@ -154,7 +154,7 @@ <h2 id="intro">Parameter Efficient Pre-training (PEPT)</h2>
 
 <h2 id="relora">ReLoRA: High-Rank Training Through Low-Rank Updates</h2>
 
-	<p>ReLoRA uses LoRA (Hu et al., 2022) decomposition technique where the pre-trained model weights are frozen and trainable rank decomposition matrices (W<sub>A</sub>, W<sub>B</sub>) are injected into each layer of the LLM. However in LoRA, the rank of the matrix is restricted by the rank r (given below), and the new trainable parameters (W<sub>A</sub> and W<sub>B</sub>) are merged back to the original matrices only after the end of the training. This suggests the potential to use PEPT techniques to pre-train LLMS. </p>
+	<p>ReLoRA uses LoRA (Hu et al., 2022) decomposition technique where the pre-trained model weights are frozen and trainable rank decomposition matrices (W<sub>A</sub>, W<sub>B</sub>) are injected into each attention and MLP layer of the LLM. However in LoRA, the rank of the matrix is restricted by the rank r (given below), and the new trainable parameters (W<sub>A</sub> and W<sub>B</sub>) are merged back to the original matrices only after the end of the training. </p>
 
 	<figure>
 		<img src="/blog/assets/images/relora-rank-property.png" />