Skip to content

Commit

Permalink
Accelerate Unsafe CAS Intrinsics on Power and X
Browse files Browse the repository at this point in the history 
Adds support for the following recognized methods:
CompareAndExchangeObject     //JDK11
CompareAndExchangeReference  //JDK17,21
CompareAndExchangeInt        //JDK11,17,21
CompareAndExchangeLong       //JDK11,17,21

Similar to their CompareAndSet counterparts, the JIT acceleration does
not support CAE on static fields so the createUnsafeCASCallDiamond
function was updated to also work on the CAE methods.

The accelerated CAE code was built on top of the existing accelerated
CAS support on both Power and X.

Removed recognized CAS enums from isUnsafePut. They are not Unsafe put
methods and these cases could not be triggered.

Even before my changes, setting TR_UseOldCompareAndSwapObject and
TR_DisableCASInlining would cause a crash on x. It doesn't really make
sense to set both at the same time but I fixed it anyways since it was
quick.

VMwrtbarWithoutStoreEvaluator is used for several different opcdes
include arraycopy, ArrayStoreCHK, writeBarrier and Unsafe CAS calls.
This is only used for the Unsafe CAS calls that store an object.

To differtiate Unsafe CAS calls from the other cases, there is a check
for the node being an icall. This check is no longer valid with the
introduction of support for compareAndExchangeReference. The node in
this case is an acall.

This changes the check to a generic call check. There is also a check
for the call being to a recognized method. This check now checks for
specific recognized methods which are:

sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z
jdk_internal_misc_Unsafe_compareAndExchangeObject
jdk_internal_misc_Unsafe_compareAndExchangeReference

Signed-off-by: jimmyk <[email protected]>
  • Loading branch information
@IBMJimmyk
IBMJimmyk committed Sep 23, 2024
1 parent dc9157d commit fda8a6f
Show file tree
Hide file tree
Showing 6 changed files with 405 additions and 122 deletions.
8 changes: 6 additions & 2 deletions runtime/compiler/codegen/J9CodeGenerator.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -655,13 +655,17 @@ J9::CodeGenerator::lowerTreesPreChildrenVisit(TR::Node *parent, TR::TreeTop *tre
// Hiding compressedref logic from CodeGen doesn't seem a good practise, the evaluator always need the uncompressedref node for write barrier,
// therefore, this part is deprecated. It'll be removed once P and Z update their corresponding evaluators.
static bool UseOldCompareAndSwapObject = (bool)feGetEnv("TR_UseOldCompareAndSwapObject");
if (self()->comp()->useCompressedPointers() && (UseOldCompareAndSwapObject || !(self()->comp()->target().cpu.isX86() || self()->comp()->target().cpu.isARM64())))
static bool disableCASInlining = feGetEnv("TR_DisableCASInlining") != NULL;
if (self()->comp()->useCompressedPointers() && ((UseOldCompareAndSwapObject && (self()->comp()->target().cpu.isARM64() || !disableCASInlining)) || !(self()->comp()->target().cpu.isX86() || self()->comp()->target().cpu.isARM64())))
{
TR::MethodSymbol *methodSymbol = parent->getSymbol()->castToMethodSymbol();
static bool disableCAEIntrinsic = feGetEnv("TR_DisableCAEIntrinsic") != NULL;
// In Java9 Unsafe could be the jdk.internal JNI method or the sun.misc ordinary method wrapper,
// while in Java8 it can only be the sun.misc package which will itself contain the JNI method.
// Test for isNative to distinguish between them.
if ((methodSymbol->getRecognizedMethod() == TR::sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z) &&
if (((methodSymbol->getRecognizedMethod() == TR::sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z) ||
((methodSymbol->getRecognizedMethod() == TR::jdk_internal_misc_Unsafe_compareAndExchangeObject) && !disableCAEIntrinsic) ||
((methodSymbol->getRecognizedMethod() == TR::jdk_internal_misc_Unsafe_compareAndExchangeReference) && !disableCAEIntrinsic)) &&
methodSymbol->isNative() &&
(!TR::Compiler->om.canGenerateArraylets() || parent->isUnsafeGetPutCASCallOnNonArray()) && parent->isSafeForCGToFastPathUnsafeCall())
{
Expand Down
8 changes: 5 additions & 3 deletions runtime/compiler/codegen/J9RecognizedMethodsEnum.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -362,9 +362,6 @@
sun_misc_Unsafe_compareAndSwapInt_jlObjectJII_Z,
sun_misc_Unsafe_compareAndSwapLong_jlObjectJJJ_Z,
sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z,
sun_misc_Unsafe_compareAndExchangeInt_jlObjectJII_Z,
sun_misc_Unsafe_compareAndExchangeLong_jlObjectJJJ_Z,
sun_misc_Unsafe_compareAndExchangeObject_jlObjectJjlObjectjlObject_Z,

sun_misc_Unsafe_putBoolean_jlObjectJZ_V,
sun_misc_Unsafe_putByte_jlObjectJB_V,
Expand Down Expand Up @@ -459,6 +456,11 @@
sun_misc_Unsafe_allocateInstance,
sun_misc_Unsafe_allocateUninitializedArray0,

jdk_internal_misc_Unsafe_compareAndExchangeInt,
jdk_internal_misc_Unsafe_compareAndExchangeLong,
jdk_internal_misc_Unsafe_compareAndExchangeObject,
jdk_internal_misc_Unsafe_compareAndExchangeReference,

jdk_internal_misc_Unsafe_copyMemory0,
jdk_internal_loader_NativeLibraries_load,
jdk_internal_util_ArraysSupport_vectorizedMismatch,
Expand Down
33 changes: 26 additions & 7 deletions runtime/compiler/env/j9method.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2975,10 +2975,10 @@ void TR_ResolvedJ9Method::construct()
{x(TR::sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z, "compareAndSetObject", "(Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;)Z")},
{x(TR::sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z, "compareAndSetReference", "(Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;)Z")},

{x(TR::sun_misc_Unsafe_compareAndExchangeInt_jlObjectJII_Z, "compareAndExchangeInt", "(Ljava/lang/Object;JII)I")},
{x(TR::sun_misc_Unsafe_compareAndExchangeLong_jlObjectJJJ_Z, "compareAndExchangeLong", "(Ljava/lang/Object;JJJ)J")},
{x(TR::sun_misc_Unsafe_compareAndExchangeObject_jlObjectJjlObjectjlObject_Z, "compareAndExchangeObject", "(Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;")},
{x(TR::sun_misc_Unsafe_compareAndExchangeObject_jlObjectJjlObjectjlObject_Z, "compareAndExchangeReference", "(Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;")},
{x(TR::jdk_internal_misc_Unsafe_compareAndExchangeInt, "compareAndExchangeInt", "(Ljava/lang/Object;JII)I")},
{x(TR::jdk_internal_misc_Unsafe_compareAndExchangeLong, "compareAndExchangeLong", "(Ljava/lang/Object;JJJ)J")},
{x(TR::jdk_internal_misc_Unsafe_compareAndExchangeObject, "compareAndExchangeObject", "(Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;")},
{x(TR::jdk_internal_misc_Unsafe_compareAndExchangeReference, "compareAndExchangeReference", "(Ljava/lang/Object;JLjava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;")},

{x(TR::sun_misc_Unsafe_staticFieldBase, "staticFieldBase", "(Ljava/lang/reflect/Field;)Ljava/lang/Object")},
{x(TR::sun_misc_Unsafe_staticFieldOffset, "staticFieldOffset", "(Ljava/lang/reflect/Field;)J")},
Expand Down Expand Up @@ -4981,6 +4981,11 @@ TR_ResolvedJ9Method::setRecognizedMethodInfo(TR::RecognizedMethod rm)
// from bool TR::TreeEvaluator::VMinlineCallEvaluator(TR::Node *node, bool isIndirect, TR::CodeGenerator *cg)
//case TR::sun_misc_Unsafe_copyMemory:

case TR::jdk_internal_misc_Unsafe_compareAndExchangeInt:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeLong:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeObject:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeReference:

case TR::sun_misc_Unsafe_loadFence:
case TR::sun_misc_Unsafe_storeFence:
case TR::sun_misc_Unsafe_fullFence:
Expand Down Expand Up @@ -5521,9 +5526,26 @@ TR_ResolvedJ9Method::isJITInternalNative()
bool
TR_J9MethodBase::isUnsafeCAS(TR::Compilation * c)
{
/*
* The TR::Compilation parameter "c" is sometimes null. But, it is needed to perform a platform target check.
* So, if it is null, this code goes and gets comp.
*/
if (nullptr == c)
{
c = TR::comp();
}

TR::RecognizedMethod rm = getRecognizedMethod();
switch (rm)
{
case TR::jdk_internal_misc_Unsafe_compareAndExchangeInt:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeLong:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeObject:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeReference:
{
TR_ASSERT_FATAL(c, "comp should not be NULL");
return (c->target().cpu.isPower() || c->target().cpu.isX86());
}
case TR::sun_misc_Unsafe_compareAndSwapInt_jlObjectJII_Z:
case TR::sun_misc_Unsafe_compareAndSwapLong_jlObjectJJJ_Z:
case TR::sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z:
Expand Down Expand Up @@ -5814,9 +5836,6 @@ TR_J9MethodBase::isUnsafePut(TR::RecognizedMethod rm)
{
switch (rm)
{
case TR::sun_misc_Unsafe_compareAndSwapInt_jlObjectJII_Z:
case TR::sun_misc_Unsafe_compareAndSwapLong_jlObjectJJJ_Z:
case TR::sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z:
case TR::sun_misc_Unsafe_getAndAddInt:
case TR::sun_misc_Unsafe_getAndAddLong:
case TR::sun_misc_Unsafe_getAndSetInt:
Expand Down
49 changes: 41 additions & 8 deletions runtime/compiler/optimizer/InlinerTempForJ9.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -403,18 +403,41 @@ TR_J9InlinerPolicy::alwaysWorthInlining(TR_ResolvedMethod * calleeMethod, TR::No
case TR::java_nio_ByteOrder_nativeOrder:
return true;

// In Java9 the following enum values match both sun.misc.Unsafe and
// jdk.internal.misc.Unsafe The sun.misc.Unsafe methods are simple
// wrappers to call jdk.internal impls, and we want to inline them. Since
// the same code can run with Java8 classes where sun.misc.Unsafe has the
// JNI impl, we need to differentiate by testing with isNative(). If it is
// native, then we don't need to inline it as it will be handled
// elsewhere.
case TR::jdk_internal_misc_Unsafe_compareAndExchangeInt:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeLong:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeReference:
if (comp()->target().cpu.isPower() || comp()->target().cpu.isX86())
{
return false;
}
break;

/* In Java9 the compareAndSwap[Int|Long|Object] and copyMemory enums match
* both sun.misc.Unsafe and jdk.internal.misc.Unsafe. The sun.misc.Unsafe
* methods are simple wrappers to call jdk.internal impls, and we want to
* inline them. Since the same code can run with Java8 classes where
* sun.misc.Unsafe has the JNI impl, we need to differentiate by testing
* with isNative(). If it is native, then we don't need to inline it as it
* will be handled elsewhere.
*
* Starting from Java12, compareAndExchangeObject was changed from being a
* native to being a simple wrapper to call compareAndExchangeReference.
* The enum matches both cases and we only want to force inlining on the
* non-native case. If the native case reaches here, it means it already
* failed the isInlineableJNI check and should not be force inlined.
*/
case TR::jdk_internal_misc_Unsafe_compareAndExchangeObject:
if (comp()->target().cpu.isPower() || comp()->target().cpu.isX86())
{
return !calleeMethod->isNative();
}
break;
case TR::sun_misc_Unsafe_compareAndSwapInt_jlObjectJII_Z:
case TR::sun_misc_Unsafe_compareAndSwapLong_jlObjectJJJ_Z:
case TR::sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z:
case TR::sun_misc_Unsafe_copyMemory:
return !calleeMethod->isNative();

default:
break;
}
Expand Down Expand Up @@ -1704,7 +1727,7 @@ TR_J9InlinerPolicy::createUnsafeMonitorOp(TR::ResolvedMethodSymbol *calleeSymbol
}

bool
TR_J9InlinerPolicy::createUnsafeCASCallDiamond( TR::TreeTop *callNodeTreeTop, TR::Node *callNode)
TR_J9InlinerPolicy::createUnsafeCASCallDiamond(TR::TreeTop *callNodeTreeTop, TR::Node *callNode)
{
// This method is used to create an if diamond around a call to any of the unsafe compare and swap methods
// Codegens have a fast path for the compare and swaps, but cannot deal with the case where the offset value passed in to a the CAS is low tagged
Expand Down Expand Up @@ -2450,6 +2473,7 @@ TR_J9InlinerPolicy::inlineUnsafeCall(TR::ResolvedMethodSymbol *calleeSymbol, TR:
!comp()->fej9()->traceableMethodsCanBeInlined()))
return false;

static bool disableCAEIntrinsic = feGetEnv("TR_DisableCAEIntrinsic") != NULL;
// I am not sure if having the same type between C/S and B/Z matters here.. ie. if the type is being used as the only distinguishing factor
switch (callNode->getSymbol()->castToResolvedMethodSymbol()->getRecognizedMethod())
{
Expand Down Expand Up @@ -2615,6 +2639,15 @@ TR_J9InlinerPolicy::inlineUnsafeCall(TR::ResolvedMethodSymbol *calleeSymbol, TR:
case TR::sun_misc_Unsafe_objectFieldOffset:
return false; // todo

case TR::jdk_internal_misc_Unsafe_compareAndExchangeInt:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeLong:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeObject:
case TR::jdk_internal_misc_Unsafe_compareAndExchangeReference:
if (disableCAEIntrinsic || !(comp()->target().cpu.isPower() || comp()->target().cpu.isX86()))
{
break;
}
// Fallthrough if previous if condition is not met.
case TR::sun_misc_Unsafe_compareAndSwapInt_jlObjectJII_Z:
case TR::sun_misc_Unsafe_compareAndSwapLong_jlObjectJJJ_Z:
case TR::sun_misc_Unsafe_compareAndSwapObject_jlObjectJjlObjectjlObject_Z:
Expand Down
Loading

0 comments on commit fda8a6f

Please sign in to comment.