-
Notifications
You must be signed in to change notification settings - Fork 728
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JDK8 FIPS cmdLineTester_CryptoTest_0_FAILED Success condition was not found: [Output match: using native crypto library] #16902
Comments
JDK 8 x86-64_linux_fips 0.38 m2 build
|
@jasonkatonica this test does some simple crypto [1] with [1] https://github.com/eclipse-openj9/openj9/blob/master/test/functional/cmdLineTests/openssl/src/org/openj9/test/openssl/CryptoTest.java |
Hello @pshipton, When executing with the flag I do not expect
@taoliult and @WilburZjh , let us know if you have thoughts otherwise :) |
So the conclusion is the failure is expected. I'm not sure why we didn't see it in previous releases, but not sure it's worth trying to figure it out. My remaining question is why it's passing on jdk17. Internal build - rhel8x64-fips3-1
|
If you are not familiar with cmdlinetester, the conditions for passing/failing are Passing if both |
Hi @pshipton , Im investigating the exceptions, i will update shortly. |
It is worrisome that the Java 17 build is passing here. I would not expect OpenSSL to be driven there either in FIPS mode. |
For reference, this is the output when running the test on a non-FIPS machine.
|
I enabled FIPS on my own machine and get the following in FIPS mode.
|
I should have mentioned the previous was with an older open source head stream build.
Using Semeru M2 I get the following.
ibmruntimes/openj9-openjdk-jdk17@7c20b28...267a5a7 One difference is the Java Restricted Security Mode The Milestone 1 build seems fine.
|
@pshipton |
This CryptoTest is quite simple, just get the cipher, initialize it using the secret key spec.
The different between the FIPS and non-FIPS is, in FIPS mode, all above come from the FIPS provider SunPKCS11-NSS-FIPS. In non-FIPS mode, all the above come from the provider SunJCE or SunJSSE. For using the native library, I searched the keyword “NativeCrypto.isAlgorithmEnabled” in OpenJDK codes, we can see, the CipherCore.java, PKCS12PBECipherCore.java, SunJCE.java, SunEntries.java, RSACore.java and SunEC.java, are using the Native library. And in the FIPS mode, we limit the providers, only allow SunEntries.java, SunPKCS11.java, SunJSSE.java and SunEC.java. So, when running the above tests, if the first provider get by Cipher.getInstance() from the provider list, is the SunPKCS11, then the above code will run without initialize provider Sun or SunEC. Then in the output, you won’t see any debug output, saying “using native crypto library”. But, if the first provider get from the provider list, is not SunPKCS11, for example, the first one get is SunEC, and then SunPKCS11, but since SunEC returned, so it will be initialized, and due to SunEC can use Native library, so, in the debug output, we will see “using native crypto library”. Following are the example, when I run the same command using the same JDK11, but different output. This one, SunEC be returned from the provider list before SunPKCS11. So SunEC static final variable will be initialized, code below. And then, even the above test code will using the SunPKCS11 provider, but we will still see the following “using native crypto library” from the output, because those are the output when initializing provider SunEC.
And the below one, SunPKCS11 return as the first provider and this provider can be used for the Cipher, then the above test code will be executed, and no SunEC needed, so SunEC won’t be initialized. So, we won’t see any "using native crypto library" output. [root@unbolts1 openj9-crypto-tao]# /root/tao/JDKs/OpenJ9-JDK11-x86-64_linux-20230330-165936/bin/java -Dsemeru.fips=true -Djdk.nativeCryptoTrace=true CryptoTest I run the same command using the same JDK11 couple of times, the output below:
|
The inconsistent behavior is not caused by AOT (shared classes) or the JIT because I can duplicate it with |
I don't think it related to AOT (shared classes) or the JIT. Currently, FIPS mode doesn't use the native crypto library. But sometime, if we see the below output, that is because, when initialized the provider SunEC, its static final variable "useNativeEC" will be initialized. And then, the debug trace from "NativeCrypto.java" will be printed out due to "jdk.nativeCryptoTrace" is true.
But, even the "useNativeEC" is true and native crypto library is used in SunEC, the algorithms FIPS mode allowed in SunEC, are still not using the native crypto. |
We should figure out why the behavior differs from run to run. We do have Attach API which initializes itself in another thread, but using |
About the behaviour, sometime when running the CryptoTest, both SunEC and SunPKCS11 are loaded, and because the SunEC will check the native library, so the “using native crypto library” will be printed out. Sometime when running the CryptoTest, only SunPKCS11 is loaded, SunEC is not loaded, so there is no “using native crypto library” printed out. The root cause is due to, loading the non java.based provider using ServiceLoader. When loading the provider using ProviderConfig.java getProvider() method, the java.base provider, like Sun, SunRsaSign, SunJCE, etc. will be loaded directly by calling their constructor, so the order defined in the java.security file will be the order when actually loading them. All the other non java.base providers, like SunPKCS11, SunEC, etc. will be loaded by the ServiceLoader.java. And in this service loader, it will iterator over the provider class from the class loader, by calling the provider class constructor and then return the provider name. If the provider name matches the needed one, then return it. But before return the needed one, other providers class constructor maybe invoked for the purpose of getting the provider name, for comparing with the needed provider name. And if the provider class constructor is invoked, that mean, that provider is initialized/loaded. In the case of CryptoTest, due to the FIPS mode crypto provider is limited to SunPKCS11. So, when trying to load the SunPKCS11 using ServiceLoader. The providers class return from the class loader will be initialized one by one(without any order) to get the provider name, until the provider name matches the name “SunPKCS11”. So, from the below two examples, we can see, the first example. Before get the “sun.security.pkcs11.SunPKCS11”, 4 other non java.based providers be initialized, which are SunProvider, JdkSASL, SunPCSC and XMLDSigRI. And due to they are not using the native crypto, so no “using native crypto library” printed out.
The below second example, before get the “sun.security.pkcs11.SunPKCS11”, 2 other non java.based providers be initialized, which are XMLDSigRI and SunEC. And due to the SunEC is using the native crypto, so there is “using native crypto library” printed out.
So, this explained, why when running the CryptoTest, sometime we can see “SunEC native crypto implementation enabled”, and sometime, there is no such output. Because, before getting this non java.based provider "SunPKCS11” in ServiceLoader from the class loader, there is no order about which provider class will be initialized before the provider class "SunPKCS11”. This is working as design and currently FIPS mode doesn’t use the native crypto. @jasonkatonica @WilburZjh FYI. |
Thanks for the explanation. It seems the best solution is to exclude the test. I also finally realized why we haven't seen this problem before, because I believe we just started running functional testing in FIPS mode. |
Nope, I checked and we were running sanity.functional before. I guess we were just getting lucky? |
I ran the test with an older version 11.018, and while SunEC is still initialized sometimes, there are no messages
|
Seems we changed the verbose output? While it used to output |
Yes, there is a PR "Lazily initialize native crypto libraries" (ibmruntimes/openj9-openjdk-jdk11@fc9cdcb) changed the message in the debug trace from JDK11 and above, but I did not see this PR in JDK8. |
The PR in question was for a CRIU issue which is not supported in JDK8 yet. |
It appears jdk8 should have always failed the test. Since it doesn't use service providers it never seems to initialize SunEC. While with jdk11+ the results vary from run to run as we have seen. |
@pshipton |
I'm not concerned about the failure any more. I still think we should just exclude the test for FIPS. |
Yes, when running the cmdLineTester_CryptoTest in JDK8, the test will always failed. From the output below, the failure reason is not because lack of output "Crypto test COMPLETED", it is because no "using native crypto library" output. So it can not meet the success conditions, which are both "using native crypto library" and "Crypto test COMPLETED" are in the output.
|
I think I finally figured out why we haven't seen this failure previously. Although we were running sanity.functional testing for past releases, it was only running a single test cmdLineTester_fips. All the other testing is being run for this release for the first time. |
Excluded cmdLineTester_CryptoTest in FIPS mode, as per the conversation found in issue eclipse-openj9#16902. Signed-off-by: Paritosh Kumar <[email protected]>
Testcase is excluded from FIPS mode as part of PR #17533. This issue can be closed. |
|
Reopening since the failure still occurs on jdk8. |
I have checked the failing built it's run again openj9 v0.40.0-release branch. Shall I ask to get it merged in v0.40.0-release? |
Yes, pls open a PR against the v0.40.0-release branch. |
JDK8 x86-64_linux_fips140_2 0.41 milestone 1(
50x internal grinder - all passed |
It seems that |
We are using test material from @renfeiw could you back port cd4ce67 into |
PR created: #18112 |
Failure link
From an internal build(
rhel8x64-fips1-1
):Rerun in Grinder - Change TARGET to run only the failed test targets.
Optional info
Failure output (captured from console output)
50x internal grinder - all failed
The text was updated successfully, but these errors were encountered: