linear_int4_kernel for XPU #1130
Conversation
Reset to bfdbaf4 --------- Co-authored-by: mengfei25 <[email protected]> Co-authored-by: LuFengqing <[email protected]> Co-authored-by: Ratnam Parikh <[email protected]> Co-authored-by: Feng Yuan <[email protected]>
| auto aptr = A; | ||
| auto cptr = C + g_n; | ||
| if constexpr (std::is_same_v<scalar_t, sycl::half>) { | ||
| sycl::half2 tmpAcc = {0.f, 0.f}; |
There was a problem hiding this comment.
is it safe to use half as acc type?
usually, the acc type for both float16 and bfloat16 are float32
| *cptr = sum[0] + sum[1]; | ||
| } | ||
| } else { | ||
| scalar_t tmpAcc = 0.f; |
There was a problem hiding this comment.
you need to be VERY careful about the acc type. Slight difference between CUDA may lead to accuracy errors that are very very difficult to debug in a finla e2e model, especially in LLM
| for (int ikk = 0; ikk < TileK; ikk += 2) { | ||
| sycl::half2 tmpA = *(sycl::half2*)&aptr[sg_id * TileK + ikk]; | ||
| sycl::half2 tmpB = { | ||
| static_cast<int8_t>((tmps8[ikk / 2] & 0x0f) - 8), | ||
| static_cast<int8_t>((tmps8[ikk / 2] >> 4) - 8)}; | ||
| tmpAcc += tmpA * tmpB * scale; |
There was a problem hiding this comment.
is it possible to do vectorized load and shift with sycl? i don't know.
if not, i guess this is best perf that we can get so far. this line should be the major bottlenecks.
| zero_points = torch.Tensor([8]).to(torch.int8).to("xpu") | ||
| weight_ba = weight.transpose(0, 1).contiguous() | ||
|
|
||
| out_onednn =torch._weight_int4pack_mm_with_scales_and_zeros( |
There was a problem hiding this comment.
a more general question is that where are we placing _weight_int4pack_mm_with_scales_and_zeros, pytorch does not have this right now.
There was a problem hiding this comment.
will be added in pytorch/pytorch#137566
| ) | ||
|
|
||
| # check gemm + bias + gelu | ||
| out_onednn_gelu = torch._weight_int4pack_mm_with_scales_and_zeros( |
There was a problem hiding this comment.
where was the signature with "tanh" defined?
does pytorch has a packed int4 gemm with post op?
|
@liangan1 CC |
|
@sunjiweiswift for the perf benchmarking, please include other configs expect M=1. This would serve as a reference of final decision making. I expect that big M would have worse perf, but that's fine, we still need to know the numbers. |
#### Bugfix - [add lazy init for empty_xpu](#1115) - [nan propagation for soft_shrink](https://github.com/intel/torch-xpu-ops/pull/1116/files#diff-b7cb5876d000db957286c8b0e72badb2b7502402c8955334f1cc21c34c98a5b9) --------- Co-authored-by: Yu, Guangye <[email protected]> Co-authored-by: ZhiweiYan-96 <[email protected]>
sunjiweiswift
force-pushed
the
fp_zp
branch
2 times, most recently
from
faa79b7 to
5a08d2e
Compare
Pure SYCL path for. int4 gemm
Benchmark results on PVC-1100. The remaining gaps are lack of usage of 2D load.
Besides PVC, the kernel can achieve
92.7% bandwidth usage on MTL
84.7% bandwidth usage on A750