Is using <repository> references / <repository-reference> in category.xml / Tycho's <addIUTargetRepositoryReferences> considered best practice?

[sratz]

Using

• <repository-reference> in category.xml
• <addIUTargetRepositoryReferences> with Tycho (https://github.com/eclipse-tycho/tycho/blob/master/RELEASE_NOTES.md#new-option-to-filter-added-repository-references-when-assembling-a-p2-repository)

is a way to declare additional dependencies and not shipping them.

This causes p2 update sites to be added to user's installation, whenever a check-for-update operation is performed.

However, it also has some drawbacks:

1. This causes and excessive amount of update sites to accumulate in the 'Available Update Sites' preference page, e.g. this screenshot is from a standard Eclipse-for-Java installation after a single 'Check for update' which was canceled
   
   Do we really want all those update sites with explicit versions to accumulate over time? There is no 'removal mechanism' when the 'parent' update site does not refer to them anymore.
2. It can cause issues where larger p2 update sites are implicitly added and things are available during check-for-update that are not actually intended. See Please do not include JustJ JREs in your update site eclipse-wildwebdeveloper/wildwebdeveloper#938 and Installing Wild Web Developer tools forces update of JustJ and Platform to latest version eclipse-wildwebdeveloper/wildwebdeveloper#1380.
3. When using dynamic (.../latest) instead of fixed (.../vX.Y) update site URLs as dependencies, this can cause implicit, unintended updates of the IDE (also Installing Wild Web Developer tools forces update of JustJ and Platform to latest version eclipse-wildwebdeveloper/wildwebdeveloper#1380).

I feel like the usage of repository references has some caveats.

[HannesWell]

I would say that p2 should not add references from p2-repositories to the Available Softare Sites.
The references should only be considered while querying the repository but the references should not be presisted in a user Eclipse.
I'm surprised that this happens, but now that you said it, I also noticed that those references accumlate over time, but until know I assumed they are added explicitly by something like Oomph or as an explicit Update site for the product.

[mickaelistria]

You can add it to the products update sites for example

Anyone willing to change Wild Web Developer to adopt this practice is welcome to do so; we'll review/merge PRs promptly!

[merks]

I've prototyped the suggested change locally:

Unfortunately it only "moves" the problem and arguably makes it even worse.

The plugin is configured to filter the IUs available from the referenced repositories:

So if I remove the reference, the repository directly contains JustJ JREs:

So this approach will not solve the problem of updating to Java 21 (or Java 22 when that is released) but rather makes it worse by locking in whatever version of JustJ was the latest at the time of the build such that one doesn't accidentally update to that latest Java 21 but rather to an older release of Java 21.

It would be nice if Tycho could do filtering without having to add a reference to the repository itself, i.e., perhaps specifying the repository URLs in the configuration directly, but that's currently not possible. It's clear that what might be desired by WWD's product is not desirable for the EPP product.

Keeping the product repository separate from the reuseable framework components' repository would be better for the consumers of WWD's components. That is the approach I've taken for "my projects" that provide products:

https://ci.eclipse.org/cbi/view/p2RepoRelated/job/cbi.p2repo.aggregator-build/

[laeubi]

As said JustJ should never be part of the build / update site and then no need for filtering at all.

The JustJ site should only be added to the eclipse-repository project (e.g. using <repository> of layout p2) and then the product should contain the feature as a root (so it could be updated) and probably it can also add the JustJ site to the product so the user will get updates without explicitly adding JustJ ...

[merks]

It is already like this:

So I don't know how to configure things differently such that JustJ is not in the product repository and is not in a repository referenced by the product repository. I think there no way, i.e., I think a product repository is always complete, except if filtered, and it can be fixed only by referencing a repository containing the thing to be filtered. I might be wrong though and I would be happy to stand corrected. I can try out whatever you might suggest...

[merks]

FYI, I did some debugging the other day when investigating if WWD could be tweaked. The adding of repositories is done very deliberately by design where org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.remember(IRepository, String) or org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.remember(RepositoryInfo, boolean) are called whenever some repository is added. There is org.eclipse.equinox.internal.p2.metadata.repository.LocalMetadataRepository.publishRepositoryReferences() which sends out an event of type org.eclipse.equinox.internal.provisional.p2.repository.RepositoryEvent.DISCOVERED that is handled to add those repositories to the manager and are then remembered. Because all this happens merely as a side-effect of loading, Oomph repository manager attempts to clean up those additions after loading:

        URI[] finalKnownRepositories = repositoryManager.getKnownRepositories(IRepositoryManager.REPOSITORIES_ALL);
        for (URI uri : finalKnownRepositories)
        {
          if (!originalKnownRepositories.contains(uri))
          {
            repositoryManager.removeRepository(uri);
          }
        }

But it's tricky to do this for p2 in general because installing something may involve touchpoints that add repositories such that one cannot simply restore the original state...

[laeubi]

It would be nice if Tycho could do filtering without having to add a reference to the repository itself, i.e., perhaps specifying the repository URLs in the configuration directly, but that's currently not possible.

Just want to put this on a different thread to not confuse things. The reason why Tycho works this way is because it is not a general filtering / modify repository action, this is supposed to remove otherwise required items (IUs+artifacts) and instead consume them from the original source! So having filtered but not adding a reference clearly makes no sense here, because then you will create a site that is missing items that a required and you can't install any software from that site anymore!

[laeubi]

Product repositories are not filtered, so I'm a bit confused to be honest.

[merks]

Well, I'm not getting any less confused either. 😕

The repository is specified to be filtered in the pom.xml and we can see in the published repository that the filtering has happened:

This commit added the references (to the category.xml) and the filter specification (to the pom):

eclipse-wildwebdeveloper/wildwebdeveloper@09f7269

So we'd like a way to not have those reference in the repo and still have the filtering. So I was merely suggesting that given the product is specify repositories that will be available during update, it might be possible to use those for filtering instead; of course that does not work currently and maybe that makes not as much sense. Of course we'd also be happy to not have only a product repository but to have also a tools/framework repository, but I think anything we want is going involve self-service. In any case, I think at this point it seems fair to conclude that we cannot achieve this goal with tools in their current form building only a product repository. Again, I'd be happy to be wrong!

[laeubi]

In any case, I think at this point it seems fair to conclude that we cannot achieve this goal with tools in their current form building only a product repository. Again, I'd be happy to be wrong!

Maybe it is because things are getting confused from udpate-sites versus "product repository", there is no such thing like a "product repository" in Tycho.

If we look here, the WWD defines a project with packaging eclipse-repository that contains

• a epp.product
• a category.xml
• a p2.inf

So what will happen here that first all content defined will be aggregated into a repository in the target/repository and from that it also contains an execution of the materialize-products mojo what will using this repository to "materialize" a zip/tar/... for all the configured platforms... and therefore it must contain everything either directly or indirectly and the result of this project is not necessarily suitable as an generic update-site it is only suitable as an update-site for the specific product(s)!

Instead one should have two modules/products (see Structured layouts), one "site" that contains the category.xml and its result is be used as a generic update-site e.g. for consumers of WWD as a library (not referencing JustJ anywhere, possibly referencing referenced sites like used Eclipse Release and is filtered), and one "product" that contains the epp.product (and possibly p2.inf even though that is better placed in a feature or configured through the product) and just leave alone the target/repository (what is not filtered) but use the produced materialized product to deploy for download of the user. This product then will possibly reference the "site" URL+JustJ URL to allow update of the items inside the product.

That way consumers of WWD (e.g. m2e) will be prevented from "surprises" by getting product dependencies dripping in while the downloaded product can still be updating its features.

[merks]

This exactly confirms my understanding as well. So the current design used by WWD is poor for consumers and can’t be fixed by tweaking the build. The team has no plans to improve or change anything.

[laeubi]

The team has no plans to improve or change anything.

But that's nothing that can be solved by Tycho/p2 or any configuration options :-)

[laeubi]

I have now addressed the problem with "surprising sites" here:

• Remember referenced repositories as system repositories instead of user #512

with that all described issues here will vanish because even though the JustJ is referenced it will not be used to search for updates (as long as the user not adding it explicitly).