Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prof algo #1

Open
wants to merge 10 commits into
base: llama_2
Choose a base branch
from
Open

Conversation

SpiritedAwayCN
Copy link

Description of changes:

  • Add profiling step to select best algorithm for cublas GEMM kernels

Note:

  • There's no command flag to disable the profiling yet. please search NO_PROF in linear_kernel.cu and inc_multihead_self_attention.cu, you can find a commented return at the beginning of the line. Removing all these comments to enable all the returns can disable this.
  • The main modifications were findBestAlgoID() funcions in both linear_kernel.cu and inc_multihead_self_attention.cu, where the profile code were implemented.
  • QKV kernels need to be grouped, and profile codes in inc_multihead_self_attention.cu requires slight changes. Please search To Xinhao in inc_multihead_self_attention.cu, the comment can help you to easily modify the profile code.
  • Some utilization functions to be moved to a better place and maybe some of the code violates the format/style rules.

Before merging:

  • Did you update the flexflow-third-party repo, if modifying any of the Cmake files, the build configs, or the submodules?

SpiritedAwayCN and others added 10 commits August 2, 2023 08:32
* n_kv_heads in inc_mha

* .

* .

* .

* .

* .

* fix

* fix

* tensor parallelism

* change weight layout

* tensor parallelism

* merge multiquery attention into inc_mha

* llama2 70B config

* spec infer change 1

* fix.

* spec infer.

* falcon spec infer.

* fix llama 70B

* fix

* fix & cleanup

* fix

* hip rocm

* issue 908

* clean debug code.

* format.

* remove multiquery. remove warning, fix python.

---------

Co-authored-by: goliaro <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants