Code

Training script#31

Code

Merged

jaidhyani merged 150 commits intomaindelphi-suite/delphi:mainfrom training-scriptdelphi-suite/delphi:training-scriptCopy head branch name to clipboardMar 19, 2024

+2,033-38

Commits

Commits on Mar 3, 2024

• add llama2.c submodule

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  5aa939a

• rename submodule to avoid import errors

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  27e7df4

• add llama2.c wrapper

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  f9dbff2

• draft training.py

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  1b394e0

• updated draft

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  9fba402

• Adding a Mamba Class

  

  SrGonao

  authored and

  jettjaniak

  committed
  e11de88

• Moving stuff to the correct place

  

  SrGonao

  authored and

  jettjaniak

  committed
  6287121

• Not ready but idea there

  

  SrGonao

  authored and

  jettjaniak

  committed
  e42a8db

• updated training script

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  6642256

• formatting

  jettjaniak

  committed
  b9cdc78

• remove gitmodules

  jettjaniak

  committed
  294e792

• moved llama2c submodule

  jettjaniak

  committed
  c95c28b

• llama2c update

  jettjaniak

  committed
  27891b5

• fix import

  jettjaniak

  committed
  cd2c5f7

• remove unused files

  jettjaniak

  committed
  145f8aa

• rename training_old -> training

  jettjaniak

  committed
  fe834b0

• Moved Mamba

  

  SrGonao

  authored and

  jettjaniak

  committed
  f7eacd7

• Added type hinting

  

  SrGonao

  authored and

  jettjaniak

  committed
  a88be1f

• Removed not needed file

  

  SrGonao

  authored and

  jettjaniak

  committed
  0d356af

• Removed compile, amp train fp32

  

  SrGonao

  authored and

  jettjaniak

  committed
  646b3e7

• fixing black and isort

  

  SrGonao

  authored and

  jettjaniak

  committed
  b04975a

• add submodules to checkout in CI

  jettjaniak

  committed
  2b056ca

• pyproject.toml, moved isort cfg, excl. llama2c

  jettjaniak

  committed
  23f8a55

• isort: llama2c known_third_party

  jettjaniak

  committed
  f7cc6b7

• limit pytest to tests/ directory

  jettjaniak

  committed
  e863604

Commits on Mar 8, 2024

• Training_script_refactor (#54)

  Show description for a00425b
  jaidhyani

  authored
  a00425b

• It's actually a script now

  jaidhyani

  committed
  2aa2ea6

• lol copypasting

  jaidhyani

  committed
  ce4a6ca

• cleanup

  jaidhyani

  committed
  83496e2

• Adding support for config files

  jaidhyani

  committed
  30a1b30

• comments

  jaidhyani

  committed
  4098036

• flag arguments take priority over config file values

  jaidhyani

  committed
  96f2361

• comments

  jaidhyani

  committed
  584c55d

• gitignore .DS_Store file on macos

  jaidhyani

  committed
  d79d50c

• remove training.sh

  jaidhyani

  committed
  9e1e9d8

• meeting notes and tweaks

  jaidhyani

  committed
  27f5d43

• configurable device

  jaidhyani

  committed
  e542237

• Adding mamba implementation

  

  SrGonao

  authored and

  jaidhyani

  committed
  0524030

• mamba hacks, please forgive me

  jaidhyani

  committed
  29e986e

• experimenting with cuda support in gh actions

  jaidhyani

  committed
  d8831bd

• welp, that didn't work

  jaidhyani

  committed
  95d534d

• remove tokenized_chunks_dataset

  jaidhyani

  committed
  8716bf7

• separate batch ordering and torch seeds

  jaidhyani

  committed
  7352eec

• remove mamba.py

  jaidhyani

  committed
  c3e5ef7

• refactoring

  jaidhyani

  committed
  7cb4ca7

• rm TODO

  jaidhyani

  committed
  7213aa4

• refactoring

  jaidhyani

  committed
  a8f7143

• bughunt

  jaidhyani

  committed
  e038f31

Commits on Mar 9, 2024

• debugger config

  jaidhyani

  committed
  59fce94

• typing improvements and bugfixes

  jaidhyani

  committed
  734b92e

• add support for "x.y.z = val" style config

  jaidhyani

  committed
  2ed386c

• first steps towards Llama2HF support

  jaidhyani

  committed
  c928d3b

• more debugging stuff

  jaidhyani

  committed
  54b095a

• initial HF llama2 support

  jaidhyani

  committed
  fee0497

Commits on Mar 10, 2024

• debug more

  jaidhyani

  committed
  2665633

• Add support for preset model configs in script, specifying multiple config files

  jaidhyani

  committed
  2f65438

• bughunt

  jaidhyani

  committed
  4c64774

• fix beartype Callalble deprecation warning

  jaidhyani

  committed
  a9ac3dd

• rm llamaconfig json accidentally added before

  jaidhyani

  committed
  3d7711a

• asdf

  jaidhyani

  committed
  c4e69d2

• script tweaks

  jaidhyani

  committed
  656228c

• better gigaconfig defaults

  jaidhyani

  committed
  27fdc79

• debug config is now just another preset; better documentation for subfield arguments

  jaidhyani

  committed
  398f1de

• fix imports

  jaidhyani

  committed
  366b4b5

• remove upload_tokens

  jaidhyani

  committed
  d4a81e8

• Whoops. I should probably test things more before pushing them.

  jaidhyani

  committed
  5dc23e6

• cleanup

  jaidhyani

  committed
  2f1a0a4

• script tweaks

  jaidhyani

  committed
  1f37228

• added support for prioritizing configs

  jaidhyani

  committed
  adfd4b4

• refactoring (config_utils) to support notebook use

  jaidhyani

  committed
  e3b326c

Commits on Mar 11, 2024

• fix Llama2ConfigData bug in gigaconfig (use default_factory)

  jaidhyani

  committed
  551a8de

• make run_training return ModelTrainingState

  jaidhyani

  committed
  cd9a5b1

• more config_utils

  jaidhyani

  committed
  ab19879

• cleanup run_training script

  jaidhyani

  committed
  bc8b43d

• training_demo notebook (for colab)

  jaidhyani

  committed
  859ae09

• static files tweak

  jaidhyani

  committed
  fc3f021

• estimate_mfu for llama2hf

  jaidhyani

  committed
  70e82ee

• Don't break if model export not available

  jaidhyani

  committed
  697f729

Commits on Mar 12, 2024

• 100k quick config

  jaidhyani

  committed
  a8f7a4f

• torch.use_deterministic_algorithms for training

  jaidhyani

  committed
  997ec3a

• import Callable from collections.abc

  jaidhyani

  committed
  f9bd899

• Move up torch.manual_seed before calling anything in torch

  jaidhyani

  committed
  89cee7c

• add wandb to requirements

  jaidhyani

  committed
  698365f

• factor out training config package + wandb_config

  jaidhyani

  committed
  7f6c180

• unused import

  jaidhyani

  committed
  662555d

• isort

  jaidhyani

  committed
  0699026

• initial mamba support

  jaidhyani

  committed
  4007b6a

• pip install wheel

  jaidhyani

  committed
  594033e

• pip install packaging

  jaidhyani

  committed
  cc010da

• come on, mamba_ssm, get it together

  jaidhyani

  committed
  bfb28c1

• requirements-nocuda.txt for gh actions

  jaidhyani

  committed
  f85d015

Commits on Mar 13, 2024

• Merge branch 'main' into training-script

  jaidhyani

  committed
  3c0010b

Commits on Mar 14, 2024

• mv ModelTypes to constants

  jaidhyani

  committed
  4870d92

• deprecate llama2c support

  jaidhyani

  committed
  9eeb960

• clear out more llama2c stuff

  jaidhyani

  committed
  02ec1a1

• we still need max_seq_len

  jaidhyani

  committed
  6cb9d52

• factoring out optimizer params from config

  jaidhyani

  committed
  35cb7c4

Commits on Mar 15, 2024

• fix broken test

  jaidhyani

  committed
  5e10db2

• model_args overhaul

  jaidhyani

  committed
  af6c0db

• rm llama2c

  jaidhyani

  committed
  2a9d2c2

• replace DataLoader

  jaidhyani

  committed
  1225d93

• run_dir to gigaconfig; output_run_dir; fix Generator type warning

  jaidhyani

  committed
  a9a791b

• save results when training is done

  jaidhyani

  committed
  10b1a36

• save step in results

  jaidhyani

  committed
  fbfeaa5

• include architecture and priority in llama preset configs

  jaidhyani

  committed
  5a54078

• Merge branch 'training-script' into mamba_dev

  jaidhyani

  committed
  19e779f

• update training demo

  jaidhyani

  committed
  a98aa42

• mamba expectedly imports correctly

  jaidhyani

  committed
  53a6adf

• rm export_model

  jaidhyani

  committed
  0d69ede

• estimate_loss no longer depends on architecture

  jaidhyani

  committed
  e24fec4

• add combine_configs (working towards frozen config for type safety)

  jaidhyani

  committed
  b9f682c

• renaming/simplification

  jaidhyani

  committed
  13db6eb

• model_config refactor to approach type safety + frozen dataclasses

  jaidhyani

  committed
  15983a3

• rm architectures.py

  jaidhyani

  committed
  ac8fb6a

Commits on Mar 16, 2024

• new config system with type safety!

  jaidhyani

  committed
  43dd1b2

• Support for optional config types (mamba and llama)

  jaidhyani

  committed
  6a21f2c

• fix sample configs

  jaidhyani

  committed
  be2354f

• remove some unused model config args

  jaidhyani

  committed
  c128f96

• remove unused mamba.py

  jaidhyani

  committed
  28076ed

• I thought I already deleted this?

  jaidhyani

  committed
  384b140

• rename to "initialize_model_training_state"

  jaidhyani

  committed
  5f64802

Commits on Mar 17, 2024

• Support for mandatory fields in run_training

  jaidhyani

  committed
  e4ecd21

• ModelTypes

  jaidhyani

  committed
  a1a150d

• output_dir is output_dir

  jaidhyani

  committed
  ec04579

• cleaner imports

  jaidhyani

  committed
  fb0c509

• error if ModelConfig doesn't include config for chosen model type

  jaidhyani

  committed
  129587d

• no-op renames for clarity

  jaidhyani

  committed
  ab208aa

• log levels

  jaidhyani

  committed
  076cd8b

Commits on Mar 18, 2024

• shebang & chmod +x on run_training.py

  jettjaniak

  committed
  be352fe

• renamed corpus dataset

  jettjaniak

  committed
  10d718e

• removed llama2c references from pyproject.toml

  jettjaniak

  committed
  77c15ef

• removed .gitmodules

  jettjaniak

  committed
  a8d03a8

• removed scripts/upload_stories.py

  jettjaniak

  committed
  2d86063

• test wandb_utils

  jaidhyani

  committed
  bed480c

• no llama2c, no .view, no need for enforcing contigious tensors

  jaidhyani

  committed
  f168310

• Fix _unoptionalize

  jaidhyani

  committed
  6a67a02

• run_training.py --help when no args

  jaidhyani

  committed
  52745bc

Commits on Mar 19, 2024

• script improvements: model-specific args moved to their own --help; fixed preset args; set default logging to INFO

  jaidhyani

  committed
  9e759a3

• rename llama to llama2

  jaidhyani

  committed
  5dfdb94

• unused imports

  jaidhyani

  committed
  12dc6c4

• set run name from config file

  jaidhyani

  committed
  a81f9e5

• set default output_dir based on run_name

  jaidhyani

  committed
  63cb45b

• remove in-progress testing file added by mistake

  jaidhyani

  committed
  be3069d

• add huggingface config

  jaidhyani

  committed
  d15f467

• fix config json that got broken somehow

  jaidhyani

  committed
  9c27671

• save/load fix + huggingface uploading

  jaidhyani

  committed
  c259e97

• fix test that broken when renaming llama to llama2

  jaidhyani

  committed
  0a04b99

• unused import

  jaidhyani

  committed
  083cb1b

• fix validation sampling

  jaidhyani

  committed
  580d3c6

• remove eval_only

  jaidhyani

  committed
  14dc55e