Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use bdimy = 1 to WAR smem race #3423

Merged
merged 6 commits into from
Nov 20, 2024
Merged

use bdimy = 1 to WAR smem race #3423

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a9396e7
use regs for multi reduction per block to avoid smem race
liqiangxl Nov 16, 2024
2c2b7ce
Merge branch 'main' into llu/use_regs_mrpb
liqiangxl Nov 19, 2024
334456b
use bdimy=1 to war race
liqiangxl Nov 19, 2024
5528bc0
Merge branch 'main' into llu/use_regs_mrpb
liqiangxl Nov 19, 2024
39d2991
clang
liqiangxl Nov 19, 2024
076e0fe
Merge branch 'main' into llu/use_regs_mrpb
liqiangxl Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 4 additions & 10 deletions csrc/scheduler/normalization_inner_outer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -657,24 +657,18 @@ std::unique_ptr<ReductionParams> innerOuterPersistentHeuristic(
if (inner_dim_numel <= 1024) {
rparams->multiple_reds_per_blk = true;
rparams->tidx_for_outer_reduction = true;
constexpr int64_t threads_per_block_mrpb = 512;

// Step-1, InnerParams, Reduction dim: inner_vect(reuse),
// inner_batch(reuse), bdimx
iop.bdimx = ceilDiv(inner_dim_numel, iop.inner_vect * iop.inner_batch);

// Step-2, InnerParams, Iteration dim: gdimy, bdimy (in next step)
iop.gdimy =
getGdimy(iop.inner_vect, threads_per_block_mrpb, iop.inner_batch);
iop.gdimy = getGdimy(iop.inner_vect, iop.bdimx, iop.inner_batch);

// Step-3, OuterParams, Iteration dim: vectorization_factor_outer(reuse),
// bdimy, gdimy (in previous step). We prefer bdimy to be larger enough to
// cover what is left in both the outer_dim and inner_dim. However, it
// should not exceed the limitation set by threads_per_block_mrpb.
int64_t bdimy_tmp = std::max(
ceilDiv(outer_dim_numel, iop.gdimy),
ceilDiv(inner_dim_numel, iop.vectorization_factor_outer * iop.gdimy));
iop.bdimy = std::min(threads_per_block_mrpb / iop.bdimx, bdimy_tmp);
// bdimy, gdimy (in previous step).
// WAR for https://github.com/NVIDIA/Fuser/issues/3428
iop.bdimy = 1;

// Step-4, OuterParams, Reduction dim: bdimx (already done)
iop.warps_per_sm = ceilDiv(iop.bdimx * iop.bdimy, dev_prop->warpSize) *
Expand Down
Loading