Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Model] Use tanh approximation of GeLU in Gemma MLP (#2106)
This is in line with the implementation in the [transformers](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma/modeling_gemma.py#L183) library. Also, the [gemma-1.1](https://huggingface.co/google/gemma-1.1-2b-it/blob/main/config.json#L10) model config.
- Loading branch information