[Sema] Change data type in MaxDispatchGrid validation calculation to …

…uint64_t to prevent overflow (microsoft#6090)

Previously, in order to determine whether the three arguments provided
to the MaxDispatchGrid attribute exceeded the maximum product value
(which is 16,777,215 (2^24-1)), each of the three components would be
multiplied together and compared to this maximum. However, the three
arguments would be read in as unsigned ints. In certain conditions, the
multiplication would cause an overflow, which would programmatically
result in a number less than the maximum product limit, despite
mathematically being above this limit. The diagnostic should trigger in
this case, but it doesn't.
This PR changes the data type of the three arguments to unsigned long
long, which can hold the maximum possible product that can be achieved
without triggering the per-multiplicand limit of 65535, and won't
overflow in this case. Now, the compiler will trigger this diagnostic.
Regression test has been added.
Fixes microsoft#5988

tools/clang/lib/Sema/SemaHLSL.cpp

@@ -13170,7 +13170,8 @@ void ValidateDispatchGridValues(DiagnosticsEngine &Diags,
         << A.getName() << "Z" << A.getRange();
     z = 0;
   }
-  if (x * y * z > MaxProductValue)
+  uint64_t product = (uint64_t)x * (uint64_t)y * (uint64_t)z;
+  if (product > MaxProductValue)
     Diags.Report(A.getLoc(), diag::err_hlsl_dispatchgrid_product)
         << A.getName() << A.getRange();
 }

tools/clang/test/SemaHLSL/attributes/overflow_node_dispatch_grid.hlsl

@@ -0,0 +1,7 @@
+// RUN: %dxc -T lib_6_8 -verify %s
+
+[Shader("node")]
+[NodeLaunch("broadcasting")]
+[NodeDispatchGrid(65535, 65535, 65535)] // expected-error {{'NodeDispatchGrid' X * Y * Z product may not exceed 16,777,215 (2^24-1)}}
+[NumThreads(1, 1, 1)]
+void myNode() { }