Inference [WIP]#475

Closed

goliaro wants to merge 389 commits intomasterfrom inference

+92,681-11,253

This pull request is big! We're only showing the most recent 250 commits

Commits on May 15, 2023

add decoder for gpt tokenizer
goliaro
committed

Commits on May 16, 2023

Commits on May 19, 2023

Commits on May 20, 2023

[Inference][CI] - Fix GPU-CI and hip_rocmbuild tests (#731 )
goliaro
authored

Commits on May 21, 2023

[Inference] - Cleanup/refactor (#732 )
goliaro
authored

Commits on May 25, 2023

Merging duplicate functions in IncMHA, SpecIncMHA, and TreeIncNHA (#736 )

jiazhihao
and
xinhaoc
authored

Commits on May 27, 2023

Commits on May 30, 2023

Supporting mixed-precision (Spec/Tree/Normal) Incremental MultiHead Attention (#737 )

jiazhihao
and
xinhaoc
authored

Commits on May 31, 2023

Commits on Jun 1, 2023

fix
xinhaoc
committed

Commits on Jun 4, 2023

[Inference] - Add half precision & HuggingFace alignment tests + Speed tests (#749 )
goliaro
authored

Commits on Jun 8, 2023

Commits on Jun 15, 2023

Fix inference test (#767 )
goliaro
authored

Commits on Jun 16, 2023

Merge branch 'inference' into fix_spec
xinhaoc
authored

Commits on Jun 17, 2023

Commits on Jun 19, 2023

[Inference] - Fix build issues (#779 )
goliaro
authored

Commits on Jun 22, 2023

Support CPU Offload in SpecInfer (#765 )

jiazhihao
and
xinhaoc
authored

Commits on Jun 23, 2023

Merge branch 'inference' into fix_spec
zwang86
authored

Commits on Jun 25, 2023

[Inference] Tensor model parallelism (#778 )
goliaro
authored

Commits on Jun 28, 2023

Commits on Jun 30, 2023

Commits on Jul 5, 2023

Merge master branch into inference (#835 )

authored

Commits on Jul 8, 2023

Commits on Jul 10, 2023

Using AllReduce instead of Reduce + Replicate when tensor model parallelism is enabled (#813 )

authored

Commits on Jul 16, 2023

Commits on Jul 18, 2023

Merge pull request #858 from flexflow/merge_master_into_inference
goliaro
authored

Commits on Jul 21, 2023

Commits on Jul 22, 2023

Python interface for inference (part 1) (#878 )
goliaro
authored

Commits on Jul 28, 2023

Commits on Jul 30, 2023

enable tracing (#901 )
jiazhihao
authored

Commits on Jul 31, 2023

Commits on Aug 2, 2023

Python interface for inference (part 2) (#893 )
goliaro
authored

Commits on Aug 3, 2023

Support Group Attention (Llama 2) (#883 )

xinhaoc
and
goliaro
authored

Commits on Aug 4, 2023

Commits on Aug 7, 2023

fix (#916 )
xinhaoc
authored

Commits on Aug 8, 2023

Commits on Aug 13, 2023

Commits on Aug 15, 2023

update tokenizers-cpp repo
goliaro
committed

Commits on Aug 16, 2023

Commits on Aug 17, 2023

Commits on Aug 18, 2023

Update README.md
jiazhihao
authored

Commits on Aug 19, 2023

Commits on Aug 21, 2023

Commits on Aug 23, 2023

Add multinode tutorial to readthedocs (#1019 )
goliaro
authored

Commits on Aug 25, 2023

Allow FlexFlow Serve to stop when EOS token is generated (#1026 )
jiazhihao
authored

Commits on Aug 27, 2023

Commits on Aug 28, 2023

Commits on Aug 31, 2023

Build Docker images for AMD gpus (#1041 )
goliaro
authored

Commits on Sep 2, 2023

Commits on Sep 3, 2023

fix group attention issue (#1062 )

authored

Commits on Sep 6, 2023

Commits on Sep 11, 2023

bug fix
goliaro
committed

Commits on Sep 17, 2023

Fix Falcon model, inference test in CI (#1138 )
goliaro
authored

Commits on Sep 19, 2023

fix ci
goliaro
committed

Commits on Sep 20, 2023

Do not run empty kernels (num_tokens=0) (#1141 )
goliaro
authored

Commits on Sep 21, 2023

Fuse inference kernels to reduce kernel launch overhead (part 1) (#1128 )
goliaro
authored

Commits on Sep 23, 2023

Fuse inference kernels (part 2) (#1143 )
goliaro
authored

Commits on Sep 24, 2023

Commits on Sep 25, 2023

[SpecInfer] Update RequestManager (#1096 )

authored

Commits on Sep 26, 2023

Fuse inference kernels (part 3) (#1146 )
goliaro
authored

Commits on Sep 29, 2023

Commits on Sep 30, 2023

Fix pip install issues affecting some platforms (#1159 )
goliaro
authored

Commits on Oct 2, 2023

[Cleanup] - Remove obsolete stuff (#1160 )

goliaro
and
jiazhihao
authored

Commits on Oct 5, 2023

Compare flle path reliably (#1173 )
suranap
authored

Commits on Oct 8, 2023

[Tool] - Add mechanism to save operators' tensors to file (#1174 )
goliaro
authored

Commits on Oct 10, 2023

fix backward gelu, layernorm (#1187 )
xinhaoc
authored

Commits on Oct 14, 2023

Optimize attention kernel v2 1.0, use Gemm replace GemmStridedBatch (#1190 )
xinhaoc
authored

Commits on Oct 16, 2023

Allow token arrangement align with request index in batch (#1176 )
zwang86
authored

Commits on Oct 17, 2023

variable renaming (#1194 )
jiazhihao
authored

Commits on Oct 18, 2023

Add first_token_offset_in_batch to indicate the offset of the request's first token in a BatchConfig (#1197 )
jiazhihao
authored

Commits on Oct 22, 2023

Update the data layout of m->attn_heads (#1204 )

jiazhihao
and
xinhaoc
authored

Commits on Oct 23, 2023

Commits on Oct 24, 2023

Commits on Nov 5, 2023

[CI/Docs/Examples] - Replace llama with llama2 model (#1219 )
goliaro
authored

Commits on Nov 6, 2023

Fix inference tests in CI (#1225 )
goliaro
authored

Commits on Nov 9, 2023

Update the default cublas behavior when CUDA_VERSION is not specified (#1209 )
jiazhihao
authored

Commits on Nov 10, 2023

Reorder tokens in batch using based on token type (#1214 )

zwang86
and
jiazhihao
authored

Commits on Nov 15, 2023

Optimize attention kernel (#1228 )

xinhaoc
and
jiazhihao
authored

Commits on Nov 17, 2023

fix ucx against inference branch (#1230 )
eddy16112
authored

Commits on Nov 28, 2023

post ucx fixes
goliaro
committed

Commits on Dec 1, 2023

Fix tensor shapes for elementwise binary operations with broadcasting (#1234 )

soumyac1999
and
jiazhihao
authored

Commits on Dec 3, 2023

Fix attention (#1238 )
xinhaoc
authored

Commits on Dec 12, 2023

Commits on Dec 24, 2023

Fix link issue (#1247 )
xinhaoc
authored

Commits on Dec 25, 2023

init
xinhaoc
committed

Commits on Dec 26, 2023

Commits on Dec 28, 2023

bitmap+tree verify
xinhaoc
committed
fix.
xinhaoc
committed

Commits on Dec 29, 2023

Commits on Dec 31, 2023

Commits on Jan 1, 2024

Commits on Jan 2, 2024

Commits on Jan 3, 2024

Commits on Jan 8, 2024

Commits on Jan 9, 2024

fix
goliaro
committed

Commits on Jan 10, 2024

Fuse bias and relu in OPT (#1265 )
goliaro
authored

Commits on Jan 12, 2024

Commits on Jan 14, 2024

Commits on Jan 20, 2024

Better debugging/logging tools for alignment checks (#1275 )

goliaro
and
Ubuntu
authored
Fix incorrect innode being checked (#1273 )

FelixBrakel
and
goliaro
authored

Commits on Jan 26, 2024

Commits on Jan 27, 2024

Bug fixes and update Legion version (#1287 )

goliaro
and
jiazhihao
authored

Commits on Feb 5, 2024

Docs Modification for Python Usecases (#1291 )

april-yyt
and
jiazhihao
authored

Commits on Feb 22, 2024

Add support for docker machines with cuda 12.1 and cuda 12.2 (#1308 )
goliaro
authored

Commits on Mar 3, 2024

Fix NCCL tear down issue, update docker pre-build cuda version list (#1318 )
goliaro
authored

Commits on Mar 9, 2024

add expansion config param in specinfer
goliaro
committed

Commits on Mar 11, 2024

parametrize max_spec_tree_token_num
goliaro
committed

Commits on Mar 13, 2024

fix
goliaro
committed

Commits on Mar 14, 2024

fix
goliaro
committed
fix
goliaro
committed

Commits on Mar 30, 2024

Commits on Apr 6, 2024

fix: 'model_configs' AttributeError (#1358 )
chenzhuofu
authored

Commits on Apr 8, 2024

Changes to support Perlmutter environment (#1360 )

goliaro
and
Gabriele Oliaro
authored

Commits on Apr 24, 2024

Commits on Apr 30, 2024

doc: fix c++ serving example (#1372 )

chenzhuofu
and
goliaro
authored

Commits on May 30, 2024

Commits on Jun 20, 2024

add code to keep runners registered
goliaro
committed

Commits on Jul 11, 2024

Commits on Aug 3, 2024

minor bug fix (#1456 )
jiazhihao
authored

Commits on Aug 12, 2024

update legion version (#1307 )
jiazhihao
authored

Commits on Aug 13, 2024

Managed mem support (#1466 )
chenzhuofu
authored

Commits on Aug 20, 2024

pip flexflow_python typo (#1461 )

stelleg
and
jiazhihao
authored

Commits on Aug 28, 2024

update legion version
goliaro
committed

Commits on Aug 31, 2024

Fix nccl-induced segfault (#1481 )
goliaro
authored

Commits on Sep 2, 2024

Fix python install issue caused by new Legion version (#1482 )
goliaro
authored

Commits on Sep 4, 2024

PEFT support (inference/finetuning) (#1153 )

authored