Bundles placed to "linked" locations not found since 4.31

[iloveeclipse]

Unfortunately I do not have (public) reproducer yet, I'm trying to create one, but it takes time.

Historically, our SDK based application consists of multiple parts and is deployed on Linux via rpm's.
Because of that:

1. we can't just make a product that could be installed into writable location and so we always start Eclipse with -configuration argument redirecting the config area to dedicated location on user home.
2. we can't not use links directory as parts of the application (like help with few GB size) are shared between different application versions installed on same workstation.

So we always built our application in a structure where we had something like

eclipseInstallRoot/
  - features
  - plugins
  - links
       linkFileOne
       linkFileTwo
       linkFileThree
 someOtherLocationForLinkOne/
  - features
  - plugins
someOtherLocationForLinkTwo/
  - features
  - plugins
etc...

So while building the application from SDK + CDT + Xtext etc + our code we did some p2 director tasks to install/uninstall things we want or don't want to see
eclipse --launcher.suppressErrors -consoleLog -nosplash -application org.eclipse.equinox.p2.director ...
and after that started our application.
eclipse -application ourApplication -configuration some_path ...

That worked till 4.31, and doesn't work anymore because the application is not found anymore on startup... We also see that bundles.info file generated after p2 director tasks doesn't contain "our" bundles from linked folders anymore.

The problem seem to be related to both application bundle located under "linked" directory and redirection of OSGI configuration area via -configuration argument.
So it seem to me that something prevents OSGI (or p2 or extension registry at the and) to find the application contribution in our case.

If the problem is a bug (and not a build / configuration issue in our code), it is a big problem to similar Eclipse based applications...

Does anyone has any pointer what is changed in p2 / OSGI area in 4.31 that could be related?

[iloveeclipse]

OK, listening to #456 I've stumbled over the sentence in https://wiki.eclipse.org/Equinox/p2/Getting_Started#Dropins that

the links folder is supported by p2 with the same behavior. This is really just a subset of the functionality available in the Dropins folder.

So I've started the p2.director task we execute with options file containing org.eclipse.equinox.p2.core/debug=true org.eclipse.equinox.p2.core/reconciler=true and see now following:

!MESSAGE Unable to satisfy dependency from <our bundle> 8.7.0 to osgi.ee; (&(osgi.ee=JavaSE)(version=21)).

... and that for every bundle we have ...

which results in

 [reconciler] [plan] Some units will not be installed, because they are already installed or there are dependency issues

So the fun is: p2 director and the application are running on Java 21 and they run without any issues on Eclipse 4.30.

So p2.director that works perfectly fine with this line in MANIFEST.MF on 4.30 is somehow broken in 4.31 :

Bundle-RequiredExecutionEnvironment: JavaSE-21

Any ideas why???

[laeubi]

The Java-SE usually is not taken from the running java but from the profile or update site, so you should make sure that your product contains Java-21 or it could be installed (e.g. JustJ Updatesite)

[iloveeclipse]

you should make sure that your product contains Java-21

Why should we add Java that is installed on every workstation to the product?

or it could be installed (e.g. JustJ Updatesite)

Same here.
We have environment we fully control, all is installed via rpm's in versions we need, including Java 21.
Java 21 is in PATH, in JAVA_HOME and Eclipse runs on it - what else is missing for PDE or p2?

[laeubi]

Why should we add Java that is installed on every workstation to the product?

I didn't wrote you should do that ...

We have environment we fully control, all is installed via rpm's in versions we need, including Java 21.
Java 21 is in PATH, in JAVA_HOME and Eclipse runs on it -

... but as said this is mostly irrelevant to P2 as it requires a unit (usually some java.jre.se...) that claims your product is running on Java (whatever).

what else is missing for PDE or p2?

dropins is a p2 feature and it complains that your product does not contain any unit (==metadata) that fulfills osgi.ee; (&(osgi.ee=JavaSE)(version=21)) those units are usually installed as part of the initial product install or from an update-site containing them (e.g. Tycho will provide one derived from your build requirements), but without more details its hard to tell, so it seems neither your product nor your update-site contain such unit but the bundle require it.

[iloveeclipse]

Sorry, I'm not p2 expert.
We just take full SDK and install extra things to it. That worked fine with 4.30.
So why do we need to add some support for Java 21 in 4.31 if 4.30 had no issues with Java 21?

[iloveeclipse]

Could you provide any pointer what piece of software on which update site contains the magic "Java 21 unit"?
Second question: why is this not included (anymore?) in 4.31?

PS: JustJ Updatesite is not an answer, as we don't use any JustJ artifacts because we use our own Java builds.

[laeubi]

Could you provide any pointer what piece of software on which update site contains the magic "Java 21 unit"?

it is usually at the "producer" site so the one that produces your site from where you install the items into your product and/or bundles to publish so one needs to know how these are produced, configured and so on, e.g. you can look at the

eclipse.platform.releng.aggregator/eclipse.platform.releng.tychoeclipsebuilder/eclipse.platform.repository/target/repository

build result as described here and search in content.xml for "a.jre.javase" you will maybe see java 17 included among others there (depending on the configured jre on your system and used for a build).

Second question: why is this not included (anymore?) in 4.31?

It depends a bit actually this is nothing you should expect from any platform release site as platform using Java-17, it is possible that it contains more units but that neither mandatory nor guaranteed.

[iloveeclipse]

it is usually at the "producer" site so the one that produces your site from where you install the items into your product

Ugh. We don't have any site, we modify SDK via p2 director (by adding/removing bundles) and ship the result via rpm.

So what exact piece of software should we produce now to make p2 happy? A bundle? A feature? Just any bundle saying "here I'm the Java 21 unit looking for" via some p2.inf?

[laeubi]

Ugh. We don't have any site, we modify SDK via p2 director (by adding/removing bundles) and ship the result via rpm.

You should have a site that you pass to p2director (e.g. -repository parameter) or what is the source of bundles to install?

So what exact piece of software should we produce now to make p2 happy? A bundle? A feature? Just any bundle saying "here I'm the Java 21 unit looking for" via some p2.inf?

This is usually metadata only so no physical file, P2 contains the JREAction to generate such units so one might can even produce something like this as a standalone it is just very unusual, therfore I think you should check if it can't be created at the sources.

For example P2 ships with the UpdateSite Publisher Application, that has a parameter -addJREIU this then should generate sich IU from the running Java but it all depends a bit on the toolset you are using.

For example if you are using Features And Bundles Publisher Application or p2.publish.featuresAndBundles ant task, this currently seems not supporting that but its quite easy to add such an option (if one don't need site/category xml)

[iloveeclipse]

For example P2 ships with the UpdateSite Publisher Application, that has a parameter -addJREIU this then should generate sich IU from the running Java but it all depends a bit on the toolset you are using.

So running that application from command line results in Java 17 added (even if I run on Java 21) => #461

This is usually metadata only so no physical file, P2 contains the JREAction to generate such units so one might can even produce something like this as a standalone it is just very unusual, therfore I think you should check if it can't be created at the sources.

Created at which sources? I have SDK & run some p2 tasks via command line.
Which source should be changed to make p2 happy and see Java 21 as a.jre.javase?

What also wonders me: in the SDK itself, as it delivered via tar.gz file, which part / p2 config contains something related to that a.jre.javase IU?

I don't see it (I've grepped). Only SDK 4.30 p2 repository contains a.jre.javase = 21 in content.xml file (and 4.31 contains 17?) but we don't use SDK p2 repository in our "offline" build (I hope so, at least we never explicitly pointed to any remote update site, only to local one with our own code), yet the build works fine with 4.30 and doesn't with 4.31.

[laeubi]

So running that application from command line results in Java 17 added (even if I run on Java 21) => #461

That's possible, I just looked through P2 what actually ever will publish a JRE Unit, so this either needs another option or rethink if always using the current JVM a fixed one sounds strange as if your are running with Java 11 it obviously can't generate one of Java 17 from it.

Created at which sources? I have SDK & run some p2 tasks via command line.
Which source should be changed to make p2 happy and see Java 21 as a.jre.javase?

Nowadays the answer is that usually Tycho is doing that for you as I mentioned here theoretically there are also ant task but obviously no one cares to maintain / enhance them to catch up with the faster developing cycles, new features are testing with other than current platform Java versions.

What also wonders me: in the SDK itself, as it delivered via tar.gz file, which part / p2 config contains something related to that a.jre.javase IU?

It is not configured in any way, Tycho can derive everything automatically, and it will add configured (== toolchains.xml for example) JREs and at the minimum the one you are running with, e.g go here and search for a.jre.javase you will see:

[2024-02-12T10:05:35.967Z] [INFO] --- tycho-p2-publisher:4.0.6-SNAPSHOT:publish-osgi-ee (default-publish-osgi-ee) @ equinox-sdk ---

[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 11.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 12.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 13.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 14.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 15.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 16.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 17.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 18.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 19.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 21.0.0]
[2024-02-12T10:05:35.967Z] [INFO] Published profile IUs: [a.jre.javase 22.0.0]

I don't see it (I've grepped). Only SDK 4.30 p2 repository contains a.jre.javase = 21 in content.xml file (and 4.31 contains 17?) but we don't use SDK p2 repository in our "offline" build (I hope so, at least we never explicitly pointed to any remote update site, only to local one with our own code), yet the build works fine with 4.30 and doesn't with 4.31.

Its hard to tell I'm still not sure what is then used, the SDK.zip (which one?), if yes it might only contain what is required for that product but I never checked that.

[iloveeclipse]

OK, it means tycho-p2-publisher step is needed to get a simple bundle installed into SDK.
We don't use maven at all, our build is offline one, just a set of shell / ant scripts & p2 tasks.

Do you mean, we need to have at least one p2 task executed by maven+tycho in order to get bundles that require Java 21 running on Java 21? That's surely an overkill to force clients use some technology just to state an "obvious" thing that Java 21 is compatible with Java 21 execution environment...

So far we could run our build fully offline and without maven.
Isn't there other possibility to generate that a.jre.javase metadata as using tycho?
And where p2 installation looks for that metadata in SDK?
It can find that somewhere in 4.30.

[iloveeclipse]

Update:

1. after fixing UpdateSitePublisher -addJREIU argument doesn't add current JVM #461
2. changing update site published to "addJREIU"
3. installing generated a.jre.javase to the target application

our bundles requiring Java 21 can be finally installed on Java 21 based application.
What an insane waste of time for absolutely nonsense work!

Why does p2 insists to see a.jre.javase 21 "installation unit" (just bunch of xml) "installed" in the platform in order to see that the JVM on which p2 is running matches perfectly the required execution environment?
Why can't p2 check if the "default environment" on which p2 runs (built on the fly from the current JRE like it is done in JREAction) can fully satisfy all needs?

Also note, I doubt this while metadata installation will work if a.jre.javasemetadata created for Java 42 and installed by p2 on Java 21 runtime will really make happy bundles expecting Java 42...

[mickaelistria]

Why can't p2 check if the "default environment" on which p2 runs (built on the fly from the current JRE like it is done in JREAction) can fully satisfy all needs?

This has been requested since the inception of p2 more or less; and no-one implemented it because it's not so trivial.

[mickaelistria]

Also note that now that more and more products happily use JustJ and don't need those a.jre units as JustJ is a p2 good citizen and makes real update path capable of updating JustJ as well, it's very unlikely that the hype grows on the topic of dynamically creating a.jre units.

[iloveeclipse]

As explained above, we do not plan to use JustJ simply because we use our evaluated, tested and patched JVM's that satisfy our needs.

If someone can point me to the place in p2 resolver that checks that a.jre.javase metadata, I would see if we can fix that with reasonable effort.

[merks]

What an insane waste of time for absolutely nonsense work!

Imagine being at the receiving end of this comment. So often we all experience frustrations, but what good comes from venting it in this way?

[iloveeclipse]

What an insane waste of time for absolutely nonsense work!

Imagine being at the receiving end of this comment. So often we all experience frustrations, but what good comes from venting it in this way?

I've literally spent a week trying to understand what in 4.31 broke our build process. That was supposed to be a no-brainer - check & fix missing dependency etc, but turned to be something saying: "here p2, install bunch of extra generated xml that you've just generated from the Java 21 you are running on if you don't trust me that you really run on Java 21".

I mean, if one can generate that xml by running p2 own JREAction, why can't the p2 resolver itself just do the same without extra xml generation & installation steps?

[merks]

I certainly understand and can relate very much to your frustration(s). I’m sure though that a long rant from me about my very long list of those frustrations would be well-received by no one and would accomplish nothing.

Indeed in a proper fully-functional design, p2 would, in an installation, synthesize an iu for the system bundle. That iu should be visible during planning. Problem solved. But of course the devil is in the details. E.g., the director might run on Java 11 but needs to create an installation that requires Java 21. It can’t know anything about Java 21 except what is recorded in some p2 repo, perhaps one with a JustJ 21 JRE, even one with a restricted subset of modules hence restricted package capabilities than a full standard JRE.

Let’s not assume this is insane nonsense just because it’s really complicated and we don’t understand it well. That it’s very frustrating is something we can all relate to.

[laeubi]

@iloveeclipse I tried to explain the technical background but seems to fail. Nothing in 4.31 broke your buildprocess, it is broken before you just didn't noticed it as you are maybe not using Java 21 before, there was some other source for a Java 21 JRE UI or whatever we simply don't know.

As Ed explained, it might be doable but no one has doing it and want to touch the very basic operation of P2 that has "worked for 20 years this way now". In fact I'm always working towards not requiring this and automate this stuff but the main issue is as we have no idea how do you produce the source repository (that is the one where the additional bundle(s) to install from) we can't give you better advice, I just gave you a hint how to workaround this, as the product is usually not the source of the JRE IU it is just installed as a consequence of it being required.

So effectively we don't know where/how "we" should improve P2 that it works for "your" process...

Beside that it really has nothing to do with JusJ, even though it is a great way to ship/update the JRE for a product.

[iloveeclipse]

Nothing in 4.31 broke your buildprocess, it is broken before you just didn't noticed it as you are maybe not using Java 21 before, there was some other source for a Java 21 JRE UI or whatever we simply don't know.

• For sure, 4.31 broke our build process.
• As already explained, the very same config works fine with 4.30 Eclipse SDK. The only changed bit is the SDK binary.
• Also as already pointed out: I see that 4.30 repository contains a.jre.javase metadata for Java 21 (in content.xml), but not 4.31. So I assume something in 4.31 is missing and with that piece missing we are unable to install bundles that require Java 21 into 4.31 SDK - while it works fine in 4.30.
• Since we don't use SDK p2 repository and don't fetch anything from Internet during the build, something in the SDK binaries itself (beside content.xml that is not there in SDK) contain traces of installed a.jre.javase metadata for Java 21, but I couldn't find it, I guess because it is encoded differently. Because I don't know what represents "installed" state of the a.jre.javase metadata for Java 21 in SDK binaries, I can't search for it in 4.31 - but it is most likely not there.
• It would be good to know where and how "installed" a.jre.javase metadata for Java 21 in SDK binaries is encoded, just to add a test for that.

So effectively we don't know where/how "we" should improve P2 that it works for "your" process...

What is needed is something like addJREIU argument in p2 publisher app added to p2 director application:
eclipse -application org.eclipse.equinox.p2.director -installIU <UI1, UI2, UI3> -addJREIU ...

so that if we install something into SDK that requires Java 21 while we are running on Java 21 p2 is able to proceed without saying Unable to satisfy dependency from <our bundle> 8.7.0 to osgi.ee; (&(osgi.ee=JavaSE)(version=21)) and simply adds the JRE profile it is running on.

If someone could point me to the right place / rough idea how to do that, I could try to add that operation to p2 director.

[laeubi]

The director is implemented by org.eclipse.equinox.internal.p2.director.app.DirectorApplication, there you will find

p2/bundles/org.eclipse.equinox.p2.director.app/src/org/eclipse/equinox/internal/p2/director/app/DirectorApplication.java

Lines 1044 to 1072 in 12f4524

	private void performProvisioningActions() throws CoreException {
	IProfile profile = initializeProfile();
	Collection<IInstallableUnit> installs = collectRoots(profile, rootsToInstall, true);
	Collection<IInstallableUnit> uninstalls = collectRoots(profile, rootsToUninstall, false);
	// keep this result status in case there is a problem so we can report it to the
	// user
	boolean wasRoaming = Boolean.parseBoolean(profile.getProperty(IProfile.PROP_ROAMING));
	try {
	updateRoamingProperties(profile);
	ProvisioningContext context = new ProvisioningContext(targetAgent);
	context.setMetadataRepositories(metadataRepositoryLocations.stream().toArray(URI[]::new));
	context.setArtifactRepositories(artifactRepositoryLocations.stream().toArray(URI[]::new));
	context.setProperty(ProvisioningContext.FOLLOW_REPOSITORY_REFERENCES, String.valueOf(followReferences));
	context.setProperty(FOLLOW_ARTIFACT_REPOSITORY_REFERENCES, String.valueOf(followReferences));
	ProfileChangeRequest request = buildProvisioningRequest(profile, installs, uninstalls);
	printRequest(request);
	planAndExecute(profile, context, request);
	} finally {
	// if we were originally were set to be roaming and we changed it, change it
	// back before we return
	if (wasRoaming && !Boolean.parseBoolean(profile.getProperty(IProfile.PROP_ROAMING))) {
	setRoaming(profile);
	}
	}
	}

If I where to implement such feature, I would go for something like:

if (addJreUI) {
	context.setExtraInstallableUnits( the generated IU );
}

you can take a look at JREAction how the IU is generated there in JREAction.createJREData(IPublisherResult)
and you dont need the artifact / descriptor, so you can asume jrelocation is always null, one most probabbly want to refactor things our there into a public static helper method as it is done for example with BundlesAction.createBundleIU(BundleDescription, IArtifactKey, IPublisherInfo)

[merks]

I’m not at a computer. If the jre iu data is in the installation it will be in the profile and is hard to search because it's in a *.gz. Somewhere nested under the p2 folder.

[laeubi]

If I where to implement such feature, I would go for something like:

Just FYI I tried this out in Tycho context and it worked like a charm, I can inject automatically generated java.se units into the director and then it does not complains that EEs are missing:

• Make the build EEs available to the materialize-products eclipse-tycho/tycho#3574

so something similar should work for P2 / Director / PDE / ... as well, just in case something bothers to implement something similar there.

[iloveeclipse]

Yes, it is still on my list.

[iloveeclipse]

Yes, it is still on my list.

=> #484

Closing this one now as the fix is merged via #486.