word_language_model: Fix Transformer init_weights

Model was not getting initialized property since it was using the decoder object instead of decoder weight to initialize zeros. Signed-off-by: Eli Uriegas <[email protected]>
sim1st · Jun 15, 2020 · 13acec6 · 13acec6
1 parent edfa9f2
commit 13acec6
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/word_language_model/model.py b/word_language_model/model.py
@@ -133,7 +133,7 @@ def _generate_square_subsequent_mask(self, sz):
     def init_weights(self):
         initrange = 0.1
         nn.init.uniform_(self.encoder.weight, -initrange, initrange)
-        nn.init.zeros_(self.decoder)
+        nn.init.zeros_(self.decoder.weight)
         nn.init.uniform_(self.decoder.weight, -initrange, initrange)
 
     def forward(self, src, has_mask=True):