A lot of bundles suddenly unpacked in product

[marschall]

Tycho Version: 4.0.4

We use Tycho to build an Eclipse product with materialize-products. We recently switched our target platform from 4.28 / 2023-06 to 4.30 / 2023-12 and suddenly most bundles are unpacked. Most notably the launcher.jar is suddenly unpacked and therefore the launcher no longer works.

Bundles now unpacked

com.sun.jna_5.13.0.v20230812-1000/
com.telekurs.pass.rcp.ssi_94.0.0.202401091451/
org.apache.felix.gogo.command_1.1.2/
org.apache.felix.gogo.runtime_1.1.6/
org.apache.felix.gogo.shell_1.1.4/
org.apache.xmlgraphics_2.9.0.v20230916-1600/
org.eclipse.core.commands_3.11.200.v20231108-1058/
org.eclipse.core.contenttype_3.9.200.v20230914-0751/
org.eclipse.core.databinding.beans_1.10.100.v20230708-0916/
org.eclipse.core.databinding.observable_1.13.100.v20230708-0916/
org.eclipse.core.databinding.property_1.10.100.v20230708-0916/
org.eclipse.core.databinding_1.13.100.v20230708-0916/
org.eclipse.core.expressions_3.9.200.v20230921-0857/
org.eclipse.core.filesystem_1.10.200.v20231102-0934/
org.eclipse.core.jobs_3.15.100.v20230930-1207/
org.eclipse.core.net_1.5.200.v20231106-1240/
org.eclipse.core.resources_3.20.0.v20231102-0934/
org.eclipse.core.runtime_3.30.0.v20231102-0719/
org.eclipse.e4.core.commands_1.1.200.v20231108-1058/
org.eclipse.e4.core.contexts_1.12.400.v20231103-2012/
org.eclipse.e4.core.di.annotations_1.8.200.v20231103-2012/
org.eclipse.e4.core.di.extensions.supplier_0.17.300.v20231028-1632/
org.eclipse.e4.core.di.extensions_0.18.100.v20231103-2012/
org.eclipse.e4.core.di_1.9.200.v20231105-0754/
org.eclipse.e4.core.services_2.4.200.v20231103-2012/
org.eclipse.e4.emf.xpath_0.4.100.v20230801-1255/
org.eclipse.e4.ui.bindings_0.14.200.v20231030-2045/
org.eclipse.e4.ui.css.core_0.14.200.v20231025-0632/
org.eclipse.e4.ui.css.swt.theme_0.14.200.v20230913-1724/
org.eclipse.e4.ui.css.swt_0.15.200.v20231025-0632/
org.eclipse.e4.ui.di_1.5.200.v20231030-2045/
org.eclipse.e4.ui.dialogs_1.4.100.v20230727-0604/
org.eclipse.e4.ui.model.workbench_2.4.100.v20230801-1255/
org.eclipse.e4.ui.progress_0.4.200.v20231030-2045/
org.eclipse.e4.ui.services_1.6.200.v20231030-2045/
org.eclipse.e4.ui.swt.win32_1.2.100.v20231030-2045/
org.eclipse.e4.ui.widgets_1.4.0.v20230317-0802/
org.eclipse.e4.ui.workbench.addons.swt_1.5.200.v20231106-1600/
org.eclipse.e4.ui.workbench.renderers.swt_0.16.200.v20231030-2045/
org.eclipse.e4.ui.workbench.swt_0.17.200.v20231030-2045/
org.eclipse.e4.ui.workbench_1.15.200.v20231030-2045/
org.eclipse.e4.ui.workbench3_0.17.200.v20231025-0632/
org.eclipse.equinox.app_1.6.400.v20231103-0807/
org.eclipse.equinox.bidi_1.4.400.v20231106-0858/
org.eclipse.equinox.common_3.18.200.v20231106-1826/
org.eclipse.equinox.console_1.4.600.v20231106-0859/
org.eclipse.equinox.event_1.6.300.v20231012-1251/
org.eclipse.equinox.http.jetty_3.9.0.v20231104-1114/
org.eclipse.equinox.http.registry_1.3.400.v20231012-1210/
org.eclipse.equinox.http.servlet_1.8.0.v20231018-1316/
org.eclipse.equinox.jsp.jasper.registry_1.2.200.v20231012-1239/
org.eclipse.equinox.jsp.jasper_1.1.800.v20231012-1238/
org.eclipse.equinox.launcher.win32.win32.x86_64_1.2.800.v20231003-1442/
org.eclipse.equinox.preferences_3.10.400.v20231102-2218/
org.eclipse.equinox.registry_3.11.400.v20231102-2218/
org.eclipse.equinox.security_1.4.100.v20231012-1825/
org.eclipse.equinox.simpleconfigurator_1.5.0.v20230919-1457/
org.eclipse.help.base_4.4.200.v20231201-0110/
org.eclipse.help.ui_4.6.100.v20231106-1240/
org.eclipse.help.webapp_3.11.200.v20231108-0716/
org.eclipse.help_3.10.200.v20231106-1240/
org.eclipse.jdt.core.compiler.batch_3.36.0.v20231114-0937/
org.eclipse.jdt.core_3.36.0.v20231115-1055/
org.eclipse.jface.databinding_1.15.100.v20230708-0916/
org.eclipse.jface.notifications_0.7.0.v20231017-1913/
org.eclipse.jface.text_3.24.200.v20231024-0828/
org.eclipse.jface_3.32.0.v20231107-0834/
org.eclipse.osgi.compatibility.state_1.2.900.v20231106-0859/
org.eclipse.osgi.services_3.11.200.v20231106-0901/
org.eclipse.osgi.util_3.7.300.v20231104-1118/
org.eclipse.osgi_3.18.600.v20231110-1900/
org.eclipse.rcp_4.30.0.v20231201-0110/
org.eclipse.swt.win32.win32.x86_64_3.124.200.v20231113-1355/
org.eclipse.swt_3.124.200.v20231113-1355/
org.eclipse.text_3.13.100.v20230801-1334/
org.eclipse.ui.forms_3.13.100.v20231102-1012/
org.eclipse.ui.workbench_3.131.0.v20231201-0512/
org.eclipse.ui_3.205.0.v20230922-1312/
org.eclipse.update.configurator_3.5.200.v20231102-0934/
org.eclipse.urischeme_1.3.100.v20230727-0604/
org.osgi.service.cm_1.6.1.202109301733/
org.osgi.service.component_1.5.1.202212101352/
org.osgi.service.device_1.1.1.202109301733/
org.osgi.service.event_1.4.1.202109301733/
org.osgi.service.metatype_1.4.1.202109301733/
org.osgi.service.prefs_1.1.2.202109301733/
org.osgi.service.provisioning_1.2.0.201505202024/
org.osgi.service.upnp_1.2.1.202109301733/
org.osgi.service.useradmin_1.1.1.202109301733/
org.osgi.service.wireadmin_1.0.2.202109301733/
org.osgi.util.function_1.2.0.202109301733/
org.osgi.util.measurement_1.0.2.201802012109/
org.osgi.util.position_1.0.1.201505202026/
org.osgi.util.promise_1.3.0.202212101352/
org.osgi.util.xml_1.0.2.202109301733/

Bundles still packaged

com.ibm.icu_74.1.0.jar
com.sun.el.javax.el_3.0.4.jar
com.sun.jna.platform_5.13.0.jar
jakarta.annotation-api_2.1.1.jar
jakarta.inject.jakarta.inject-api_2.0.1.jar
javax.annotation_1.3.5.v20200909-1856.jar
org.apache.aries.spifly.dynamic.bundle_1.3.7.jar
org.apache.batik.constants_1.17.0.v20231009-1000.jar
org.apache.batik.css_1.17.0.v20231009-1000.jar
org.apache.batik.i18n_1.17.0.v20231009-1000.jar
org.apache.batik.util_1.17.0.v20231009-1000.jar
org.apache.commons.cli_1.6.0.jar
org.apache.commons.collections_3.2.2.jar
org.apache.commons.commons-beanutils_1.9.4.jar
org.apache.commons.commons-codec_1.16.0.jar
org.apache.commons.commons-collections4_4.4.0.jar
org.apache.commons.commons-io_2.15.0.jar
org.apache.commons.jxpath_1.3.0.jar
org.apache.commons.lang3_3.13.0.jar
org.apache.commons.logging_1.2.0.jar
org.apache.felix.scr_2.2.6.jar
org.apache.logging.log4j.api_2.21.1.jar
org.apache.lucene.analysis-common_9.8.0.v20230929-1030.jar
org.apache.lucene.analysis-smartcn_9.8.0.v20230929-1030.jar
org.apache.lucene.core_9.8.0.v20230929-1030.jar
org.eclipse.emf.common_2.29.0.v20230916-0637.jar
org.eclipse.emf.ecore.change_2.15.0.v20230211-1150.jar
org.eclipse.emf.ecore.xmi_2.36.0.v20231002-1156.jar
org.eclipse.emf.ecore_2.35.0.v20230829-0934.jar
org.eclipse.equinox.launcher_1.6.600.v20231106-1826.jar
org.eclipse.jetty.ee8.security_12.0.3.jar
org.eclipse.jetty.ee8.server_12.0.3.jar
org.eclipse.jetty.ee8.servlet_12.0.3.jar
org.eclipse.jetty.http_12.0.3.jar
org.eclipse.jetty.io_12.0.3.jar
org.eclipse.jetty.security_12.0.3.jar
org.eclipse.jetty.server_12.0.3.jar
org.eclipse.jetty.servlet-api_4.0.6.jar
org.eclipse.jetty.session_12.0.3.jar
org.eclipse.jetty.util_12.0.3.jar
org.eclipse.orbit.xml-apis-ext_1.0.0.v20230923-0644.jar
org.jboss.logging.jboss-logging_3.5.3.Final.jar
org.jdom_1.1.3.v20230812-1600.jar
org.mortbay.jasper.apache-jsp_9.0.52.jar
org.objectweb.asm.commons_9.6.0.jar
org.objectweb.asm.tree.analysis_9.6.0.jar
org.objectweb.asm.tree_9.6.0.jar
org.objectweb.asm.util_9.6.0.jar
org.objectweb.asm_9.6.0.jar
slf4j.api_2.0.9.jar

[akurtakov]

Seems relate to https://eclipse.dev/eclipse/news/4.30/pde.php#unnecessary-attributes-removal but I can't understand why no one else complained yet.

[marschall]

Seems relate to https://eclipse.dev/eclipse/news/4.30/pde.php#unnecessary-attributes-removal but I can't understand why no one else complained yet.

Possible but my understanding is Tycho ignores unpack in feature.xml and instead relies on Eclipse-BundleShape see #1035

[laeubi]

@marschall can you please provide an example at best as an integration-test to demonstrate the issue?

You also wrote that

We recently switched our target platform from 4.28 / 2023-06 to 4.30 / 2023-12 and suddenly most bundles are unpacked.

so I assume Tycho version stays constant, can you try out the current 4.0.5-SNAPSHOT build if the problem persist?

[marschall]

@marschall can you please provide an example at best as an integration-test to demonstrate the issue?

I'll see what I can do. I may take some time.

so I assume Tycho version stays constant

Actually we updated from 3.0.4 to 4.0.4 but it also happens with 3.0.4.

can you try out the current 4.0.5-SNAPSHOT build if the problem persist?

Same problem.

[maxrabbi]

@marschall I can confirm I verified the same problem. I was trying to move our product platform to the latest Eclipse 4.30 (from 4.29). But I stumbled upon this.
What I can say is that it seems not Tycho dependent, meaning that the error occurred in both 3.0.5 (original one) and 4.0.6.
I also gave it a try packaging the old product based on 4.29 using Tycho 4.0.6. No problem verified.

[akurtakov]

This is most likely caused by eclipse-equinox/p2@89f12f7

[laeubi]

Also see here:

• Absence of an unpack attribute must not result in BundleShapeAdvice eclipse-equinox/p2#477
• Add test case for PublishFeaturesAndBundlesMojo marking bundle unpacked #3538
• Remove irrelevant attributes of plugin element in feature.xml eclipse-pde/eclipse.pde#730

[Fitzcarraldo75]

Unfortunately that does not solve the unpack problem for us. We still face that issue with 4.31 and 4.32.M1.

When comparing the content.xml created by tycho-p2-repository:4.0.7:assemble-repository using 4.32.M1 with the file created when using 4.29 we see that difference in the touchpoint data:

<touchpoint id='org.eclipse.equinox.p2.osgi' version='1.0.0'/> <touchpointData size='1'> <instructions size='2'> <instruction key='zipped'> true </instruction> <instruction key='manifest'> Bundle-SymbolicName: org.eclipse.equinox.launcher;singleton:=true&#xA;Bundle-Version: 1.6.800.v20240330-1250 </instruction> </instructions> </touchpointData>

vs

<touchpoint id='org.eclipse.equinox.p2.osgi' version='1.0.0'/> <touchpointData size='1'> <instructions size='1'> <instruction key='manifest'> Bundle-SymbolicName: org.eclipse.equinox.launcher;singleton:=true&#xA;Bundle-Version: 1.6.500.v20230717-2134 </instruction> </instructions> </touchpointData>

If we remove the

<instruction key='zipped'> true </instruction>

from content.xml before tycho-p2-director:4.0.7:materialize-products runs it works as desired for the equinox.launcher. No unpack.

[maxrabbi]

Hi, I thought I already reported some weeks ago that the situation did not change for me either when I tried to work with the latest 4.31.
So @Fitzcarraldo75 did you use this trick to produce the correct output like it was before (in 4.29) ?

[laeubi]

@Fitzcarraldo75 @maxrabbi can you provide an integration-test to demonstrate the issue?

[Fitzcarraldo75]

Hi there

I finally had time to look into our issue closely. It turned out that it was an issue in our infrastructure. To be more precise we upgraded to the target platform 4.31 which obviously required in our case as well a Tycho upgrade. We did that too and upgraded to Tycho 4.0.7.
So far so good. We did that on a feature branch. But we also have a pipeline for building and uploading a p2 repository tailored to our application. However that pipeline runs on our "mainline" still using an old Tycho version - namely 3.0.4. That led to "zipped = true" touchpoint entries in the content.jar of the p2 repo. That was our issue. Sorry for the confusion!

Summary: we built a p2 repo out of the 4.31 platform using an old tycho version. That caused the "unpack issue".

@maxrabbi
Maybe you did something similar?

[marschall]

I consider this fixed now.