2. Unit-Scaled Maximal Update Parameterization (u-μP)
-A library for unit scaling in PyTorch, based on the paper Unit-Scaled Maximal Update Parametrization and previous work Unit Scaling: Out-of-the-Box Low-Precision Training.
-Documentation can be found at -https://graphcore-research.github.io/unit-scaling and an example notebook at examples/demo.ipynb.
-Note: The library is currently in its beta release. -Some features have yet to be implemented and occasional bugs may be present. -We’re keen to help users with any problems they encounter.
-2.1. Installation
-To install the unit-scaling library, run:
pip install git+https://github.com/graphcore-research/unit-scaling.git
-2.2. What is unit scaling?
-For a demonstration of the library and an overview of how it works, see -Out-of-the-Box FP8 Training -(a notebook showing how to unit-scale the nanoGPT model).
-For a more in-depth explanation, consult our paper -Unit Scaling: Out-of-the-Box Low-Precision Training.
-And for a practical introduction to using the library, see our User Guide.
-2.3. Development
+2. Development
For users who wish to develop using this codebase, the following setup is required:
First-time setup:
then view docs/_build/html/index.html in your browser.
2.4. License
-Copyright (c) 2023 Graphcore Ltd. Licensed under the Apache 2.0 License.
-See NOTICE.md for further details.
-Development1.5. Optimising unit-scaled models -
- 4.1. Almost scaled dot-product self attention diff --git a/limitations.html b/limitations.html index e5b8b73..fa073b6 100644 --- a/limitations.html +++ b/limitations.html @@ -19,7 +19,7 @@ - + @@ -96,7 +96,7 @@