Support hiding some JRE packages from the plugin

[testforstephen]

Use Case:
I'm working on a project that leverages javac as both a parser and compiler to provide Java language support. Previously, our tool relied on the javac bundled with the installed JDK, requiring users to install the latest JDK to get the new feature support. To eliminate this dependency, we are trying to embed a standalone javac JAR in our plugin to provide these capabilities independently of the system JDK.

However, a challenge arises because the embedded javac JAR contains classes that overlap with those in JRE modules, such as java.compiler and jdk.compiler. In practice, the JRE's classes take precedence, so the plugin classloader prioritizes the JRE versions of these classes over those in our embedded javac.

I'm looking for guidance on whether the OSGi platform provides a way to hide specific packages on the classpath from the plugin and allow it to use its own versions of overlapping classes.

[laeubi]

@testforstephen due to technical limitations you cannot "replace" any package from the java.* namespace, but if you embedd the jar and simply don't import the package otherwise you will get it from your embedded jar (I assume its on the Bundle-Classpath...).

[merks]

JustJ provides JREs that include javac:

https://eclipse.dev/justj/?page=documentation

I wonder if you can't just reuse that.

As @laeubi suggests, I don't think your use case can be supported...

[testforstephen]

@merks @laeubi thanks for the response. Our project (JDT language server) uses javac API directly rather than invoking the javac.exe. The namespaces provided by the javac API are in the javax.* and com.sun.* packages. Currently, I’ve embedded a javac.jar in my bundle and set it on the Bundle-ClassPath in the bundle's MANIFEST.MF. Eclipse compiles this setup without issues, but at runtime, the bundle continues to use the javac API from the JRE instead of the embedded version.

If there is no existing solution for it yet, would it be possible to use a custom classloader to prioritize loading our local version over the JRE’s? Or could we extend the MANIFEST.MF to support an attribute like Hide-Classpath-Package to allow hiding packages on the classpath from the bundle?

[laeubi]

javax.* and com.sun.* should be fine, you should check the following things:

1. There is no Import-Package for these packages you want to use from the jar, otherwise they might get priority
2. The jar is included in the binary build (e.g via build.properties)
3. It could be that you need to disable delegate classlaoding with osgi.compatibility.bootdelegation=false see here but this depends on your deployment configuration.
4. If it is allowed to embed /ship javac.jar from a legal perspective

[merks]

I think you should check out what the JDT project is doing to enable using javac for building the abstract syntax tree. Maybe they already solved this problem?

[testforstephen]

javax.* and com.sun.* should be fine, you should check the following things:

1. There is no Import-Package for these packages you want to use from the jar, otherwise they might get priority
2. The jar is included in the binary build (e.g via build.properties)
3. It could be that you need to disable delegate classlaoding with osgi.compatibility.bootdelegation=false see here but this depends on your deployment configuration.
4. If it is allowed to embed /ship javac.jar from a legal perspective

@laeubi I have configured the local javac.jar path in both Bundle-ClassPath of MANIFEST.MF and bin.includes of build.properties. And in the Import-Package attribute, I didn't configure javax.* and com.sun.* packages. In Eclipse, when I navigate to javac API references in my code, it correctly links to the local javac.jar.

However, during debugging the bundle, the Java runtime prioritizes the system module when finding javac api classes. I use the Java command line parameters -verbose:class --show-module-resolution to view the class/module loading logs, I can see that the runtime loads the javac classes from the JRE’s modules (e.g. java.compiler, jdk.compiler) instead of the local javac.jar. This indicates the bundle classloader prioritizes JDK classes over local classes.

I will take a look at osgi.compatibility.bootdelegation=false to see if it helps.

I think you should check out what the JDT project is doing to enable using javac for building the abstract syntax tree. Maybe they already solved this problem?

@merks JDT is using ECJ (Eclipse Compiler for Java) to build AST, the thing we're doing is to make JDT to support an alternative parser/compiler (e.g. javac).

[merks]

I suggest you revisit what jdt is doing now because it’s doing exactly what you just stated.

[akurtakov]

I suggest you revisit what jdt is doing now because it’s doing exactly what you just stated.

@merks What you speak about is being done in subproject of jdt - jdt.ls (https://projects.eclipse.org/projects/eclipse.jdt.ls) and is not (yet!) part of main jdt. Also @testforstephen is one of the main drivers of that works and his request is in order to make this work easier.
I hope this clears the confusion.

[laeubi]

@testforstephen Its a bit hard to guess what happens, maybe you can share a small example?

In general the class-loading works like described here:

https://docs.osgi.org/specification/osgi.core/8.0.0/framework.module.html#i3174728

and as mentioned above, the only package that can't be overridden is java.* so there must be something fishy in your setup or interpretation of results.

[tjwatson]

I agree with @laeubi there must be something strange with your bundle. Like requiring the system bundle or org.eclipse.osgi. Keep in mind that using require-bundle here will cause issues. For example, using Require-Bundle for org.eclipse.core.runtime will pull in all of org.eclipse.osgi which in-turn exposes you to all the JDK packages because this horrible re-export:

https://github.com/eclipse-platform/eclipse.platform/blob/6dd67323474b1b49d541bcd1e4ea8007ab2a36a4/runtime/bundles/org.eclipse.core.runtime/META-INF/MANIFEST.MF#L12C17-L12C87

Something like that will prevent you from loading the packages from your bundle's class path.

[merks]

1. I expect it would be helpful to see concretely exactly what is currently being tried because you'll note that most of the answers are kind of theoretical based on hypotheses about what's actually being done.
2. This issue appears to be more like a GitHub discussion than an actionable GitHub issue, especially given its asking for guidance and it provides no concrete way of reproducing "a/the" problem.

[testforstephen]

@merks @laeubi @tjwatson @akurtakov thank you all for the suggestions.

Here are the reproducing steps:

• Open the sample project in Eclipse. org.eclipse.jdt.javac.experiment.zip
• Launch it as an Eclipse application. You can add -verbose:class to vmArgs to enable the class load logs.
• In the debuggee Eclipse window, click the menu Sample Menu -> Sample Command that will trigger the target javac api code.

As a result, it throws errors because javac api is loaded from jrt:/java.compiler rather than my local nb-javac.jar. You can view the class load logs in the original Eclipse Console viewlet, and search the keywords such as "javax.tools", "com.sun".

A video to show what I observed.

javac_class_loading.mp4

[mickaelistria]

The reason is that org.eclipse.core.runtime reexports all packages of org.eclipse.osgi and these import all system packages

OK, So one issue in our particular case is that we get this Require-Bundle: org.eclipse.core.runtime from the JDT fragment. @testforstephen we should investigate replacing this in JDT Core by some Import-Package then.

[laeubi]

OK, So one issue in our particular case is that we get this Require-Bundle: org.eclipse.core.runtime from the JDT fragment. @testforstephen we should investigate replacing this in JDT Core by some Import-Package then.

Please be aware that Import-Package might not work for JDT because we have split packages. Anyways often another bundle (e.g. equinox.common) can be required but this all depends heavily on why it is required (for what classes) at all.

Also it might be good to simply do the javac stuff in an own bundle (so it has an own classloader) and only communicate with a dedicated service interface.

[testforstephen]

The reason is that org.eclipse.core.runtime reexports all packages of org.eclipse.osgi and these import all system packages

OK, So one issue in our particular case is that we get this Require-Bundle: org.eclipse.core.runtime from the JDT fragment. @testforstephen we should investigate replacing this in JDT Core by some Import-Package then.

@mickaelistria I understand the reason why we make javac bundle to be a fragment of JDT Core because our javac DOM parser accesses some package-private classes from JDT Core. Do you think how possible to get rid of these package-private usages in javac bundle?

[mickaelistria]

Do you think how possible to get rid of these package-private usages in javac bundle?

I think it would be a lot of work, and it's IMO safer for JDT to not open further packages. It's IMO simpler to evaluate replacing the Require-Bundle: org.eclipse.core.runtime in JDT at this stage.
Other approaches such as making JDT open more packages, or creating some intermediary DOM Binding model as API between JDT Core and JDT.Javac and so on would take much more time with a much higher risk of failure. Trying to change the Require-Bundle seems like a low-hanging fruit compared to any other approach I can think of.

[jukzi]

Also it might be good to simply do the javac stuff in an own bundle (so it has an own classloader) and only communicate with a dedicated service interface.

+1

[testforstephen]

@tjwatson @laeubi Thank you for the suggestions. Replacing Require-Bundle with Import-Package for org.eclipse.core.runtime and org.eclipse.osgi helps resolve the re-export package issue and ensures the local JAR package takes precedence.

[mickaelistria]

Following up on this, it seems like for this case, we have the following options to do things proper from a OSGi perspective:

1. Make org.eclipse.core.runtime not reexport org.eclipse.osgi bundle requirement, or
2. Replace Require-Bundle: org.eclipse.core.runtime;bundle-version="[3.29,4.0)" in org.eclipse.jdt.core by some Import-Package

But both have serious drawbacks that make them most likely not desired:
Proposal 1. may break tons of clients (mostly those using the NLS class from the org.eclipse.osgi bundle by taking it as transitively provided by org.eclipse core.runtime); and proposal 2. looses the versioning which is very useful for API Tools (mainly because the Export-Package are not versioned themselves in their providers).

So I'm wondering, for this niche use-case, is there a good strategy (such as a system bundle, or some lower-level OSGi hooks) to override how the dependency are being read by Equinox? If so, we could maybe try to just override how the MANIFEST.MF for org.eclipse.jdt.core is read, and just do the replacement we want dynamically, without needing upstream projects to change and thus keeping decent compatibility with those.
What do you think?

[merks]

Really, there's a bundle-name too? You learn something new every day, equinox.common also versions that package.

Doesn't that eliminate the drawback of option 2 without requiring anything else to change?

[laeubi]

I think we should consider making a breaking change and risk breaking require-bundle users that are depending on this nonsensical re-export giving them access to JVM packages.

I'm all for that, do you like to propose a PR it seems a good time before M1 to do that.

[laeubi]

Really, there's a bundle-name too?

You can find all attributes of the bundle namespace here:
https://docs.osgi.org/specification/osgi.core/8.0.0/framework.namespaces.html#framework.namespaces.osgi.wiring.bundle

and for packages:
https://docs.osgi.org/specification/osgi.core/8.0.0/framework.namespaces.html#framework.namespaces.osgi.wiring.package

[mickaelistria]

Indeed, those extra attributes were also unknown to me and I believe that replacing Require-Bundle by some Import-Package: ...;bundle-symbolic-name="org.eclipse.core.runtime";bundle-version="[3.29.0,4.0.0)" seems to be the shortest path to success for our case. If it works, we'll try to push such change to JDT upstream.
But if someone wants to fix Platform well to remove the reexport once and for all, that would be great too. I just don't think our JDT/Javac-crew will handle it at the moment; but we'd be very pleased to freely benefit from someone else's effort :P

[tjwatson]

I think we should consider making a breaking change and risk breaking require-bundle users that are depending on this nonsensical re-export giving them access to JVM packages.

I'm all for that, do you like to propose a PR it seems a good time before M1 to do that.

I think some discussion and a plan needs to happen before jumping into a breaking change. An announcement to cross-project and the community would be needed IMO before charging ahead. I said we should consider doing this, not do it immediately.