Add something to support tools install with associated bndtools workspace templates

[scottslewis]

Currently, to install tooling (e.g. remote services) it's necessary to both install Eclipse plugins (e.g. for new editors, views, wizards, etc) and setup a bndtools workspace based upon a workspace template (for making available repositories, project templates, and other bndtools workspace resources).

There is a discussion of this use case in the bnd discourse here: https://bnd.discourse.group/t/workspace-templates/425

I propose adding a bndtools-specific p2 custom touchpoint so that newly installed (by p2) bundle can optionally setup...as part of the tooling install, a bndtools workspace based upon a given workspace template. This way, feature/bundle installation would create a bndtools workspace, based upon a given workspace template and make it unnecessary for the user to manually setup the correct workspace template.

An example of the usefulness of this is provided by ECF remote services. There already exists a plugin to do proto3 editing for a gRPC remote services distribution provider: https://github.com/ECF/grpc-RemoteServicesProvider/tree/master/tooling/org.eclipse.ecf.bndtools.grpc.ui. There also exists a bndtools workspace template that provides gRPC project templates, etc: https://github.com/ECF/bndtools.workspace. It would make things much easier to use if the plugin install and workspace template usage were combined into one p2 feature install. This could be easily done with a custom p2 touchpoint.

[pkriens]

I really don't get it ... it seems strange to install a tool and then get a workspace setup? Isn't the idea to get lots of workspaces set up? Not just at tool installation? In my mind the installation process of a tool should never be connected to a workspace creation? That should be something you do with the tool?

I am not against it because I just don't get the idea behind it. Can you give a demo or lay out in steps how this would work for a user?

[scottslewis]

Peter I think you should try using bndtools a little...as if you've your a junior Java programmer who's never used OSGi services before. Or maybe just watch a real developer try to use it for the first time (without being able to help them).

For an example, let's say you hear somehow that there are tools for gRPC remote services development...e.g. you watch one of the videos pointed to here https://github.com/ECF/grpc-RemoteServicesProvider. Please don't argue with the use case...more developers use gRPC for remoting services than use OSGi in toto, so it's pretty likely that a random developer will know more about/used gRPC than bndtools or OSGi Services.

Steps:

1. Get and install Eclipse
2. Get and install bndtools (7.1 isn't out yet so have to get snapshot)
3. Maybe you hear that there's a new OSGi Service Wizard (7.1) in bndtools (e.g. 7.1 readme) and you decide to try that out and create an OSGi service using the wizard because years ago you used Eclipse for Java dev and the wizards were helpful for getting started.
4. You go to New->OSGi Service->Java OSGi Service Wizard you are told that you first need to create a bndtools workspace.
5. Puzzled at first (I've already got an Eclipse workspace, what's a bndtools workspace), you create a New Bndtools Workspace using one of the built in templates...because that's all the workspace templates you see as available with bndtools 7.1.
6. Now go back to New->OSGi Service->Java OSGi Service Wizard and after filling in the project name and service name in wizard it creates three projects: api, impl, consumer. Nice. Further, you examine the classes in each of the projects, decide to start/run the wizard created service to see how everything gets wired up (ds) and called. After looking at the artifacts in the wizard-created projects you eventually figure out (?) that you have to open bndrun file (in api project), click 'Resolve' on lower right of bndrun editor to have it resolve.
7. You somehow figure out that after 6 you can run or debug the wizard-created Java service. And you do so. It produces a little console output showing that the service is registered, discovered by ds (consumer) and then the service method called. You can examine the code for the impl and/or consumer and can create breakpoints to see/understand what happens when.
8. Now...you would like to create a gRPC Remote Service not just a Java Remote Service. So you go back to the wizard...no such wizard exists.
9. Somehow you figure out that there is additional tooling needed to do gRPC remote services (gRPC Remote Services Tooling). Somehow you figure out that it's located here: https://raw.githubusercontent.com/ECF/grpc-RemoteServicesProvider/master/build/plugins (p2 repository) and you install the correct feature from this repo (the gRPC tooling feature...with a protoc editor and gRPC Service Wizard!).
10. Now you go to New->OSGi Services->Remote Services and there's a gRPC OSGi Service wizard! You select it and fill out the wizard fields and click finish.
11. The wizard fails...with either an error message or just fails...BECAUSE the bndtools workspace template (built-in) you used doesn't have the templates . Rather the workspace template that provides project templates, repositories, build setup, etc is at the workspace template here: https://github.com/ECF/bndtools.workspace
12. If the new bndtools user realizes that it's necessary to create a new bndtools workspace...that uses these templates, they will have to figure out that the Preferences->Bndtools preferences needs to be changed (to add the ECF github workspace template) then restart Eclipse to create a new workspace, create a new bndtools workspace using the ECF bndtools workspace in that new Eclipse workspace. If they don't realize it's necessary to setup/point to and create a ECF bndtools workspace then they will be stuck/done.
13. Assuming that they get through 12, and create a new bndtools workspace based upon ECF bndtools workspace template, they can then go to New->OSGi Services->Remote Services->gRPC OSGi Remote Service and the wizard will work to create the api, impl, consumer projects, the proto3->java code generator will be automatically downloaded generate the java code for the gRPC service api, impl, and proxy. Further, as per the videos the user can double click on the template-provided proto file, the tool for editing proto3 (syntax highlighting only current) can be used to edit the gRPC service declaration and then because of the proto3->java code generation the java code for the edited proto file will be updated immediately (and compiled, jared, etc) using the ECF workspace template-provided repositories. This is a pretty happy path for gRPC OSGi Remote Service development and is far better for gRPC development provided by any other tooling (because of OSGi Services/Remote Services/RSA dynamics, security, ds, distribution provider standardization, yatta, yatta).

But the tooling and templates setup for this scenario (steps 1-13) is a install and config/setup mess for new bndtools and/or bndtools-based tools. IMHO a big part of the mess is the separation between tooling install and bndtools workspace creation/configuration via workspace templates (which is the 'container' for cnf with project templates, repositories).

This issue is intended to provide an 'optional/user-selected' work-around for this mess. I understand that there is an intentional separation between workspaces (eclipse and bndtools) and tooling (eclipse functionality/code). I think it was a mistake made years ago to not connect workspace meta-data/config to install/configure/workspace setup in Eclipse but that's done...and the separation between bndtools workspaces and eclipse workspaces makes things much worse IMHO, but again that's done too.

So now, only talking about implementation of workarounds:

1. Enhance bndtools workspace creation to do tools install. This is the approach taking by the Eclipse's maven support (m2eclipse...i.e. if you use certain things in your pom.xml (certain maven plugins) the m2eclipse will run it's own UI and p2 to install the appropriate maven plugins.

2. Enhance p2/tooling install to allow for bndtools workspace creation (or just workspace enhancement...e.g. adding templates, repositories, etc).

Although I think it's debatable which is better I think 2 is probably (much) less UI work. And I have to admit I find the m2eclipse extension install sort of jarring (so to speak).

In our discourse discussion we basically both got to 2. As I see it, 2 can/could be implemented two ways: 1) A bundle that starts/runs the code you described to copy the workspace template locally (if at github) and setting up a couple of workspace config properties (in java) via Workspace classes/services. 2) As a p2 custom touchpoint...again simply copying the workspace template locally and setting up a couple of workspace config properties...basically running the same java code as 1 as part of p2's install of a specific feature or bundle that refers to this touchpoint. There are p2 extension points for both the UI (dialog with user/installer) and the actual p2 install process. ProvisioningAction is the abstract super class.

I've already created a provisioning action and am exploring it's API (e.g. when/how it gets executed during install). I'm also looking at how the UI extension point can be used to interact with the user (e.g. Create a Remote Services Workspace?).

[chrisrueger]

I cannot add anything from a technical level to this discussion. I just want mention that I like the idea to simplify the setup of things (in other words: minimizing the 13-step prose recipe as much as possible to achieve more of a 1-click setup).

I heard that some people use Eclipse Oomph (Tutorial) for these "1 click Eclipse setup" kind of things. But I am not familiar with the tool... I might be wrong, or it is overkill or maybe also not possible to do what @scottslewis wants to do. But maybe it just serves as a comparison or way to define non-goals.

[scottslewis]

I cannot add anything from a technical level to this discussion. I just want mention that I like the idea to simplify the setup of things (in other words: minimizing the 13-step prose recipe as much as possible to achieve more of a 1-click setup).

I heard that some people use Eclipse Oomph (Tutorial) for these "1 click Eclipse setup" kind of things. But I am not familiar with the tool... I might be wrong, or it is overkill or maybe also not possible to do what @scottslewis wants to do. But maybe it just serves as a comparison or way to define non-goals.

Thanks Chris. I had forgotten about Oomph as a possible solution and don't know enough about it to say whether it could provide a solution better than p2 touchpoints...that's why I mentioned p2 touchpoints in the subject line. FWIW, I'm open to using something like Oomph (and changing subject line) if it could handle my bndtools use cases (e.g. as described above) in an easier way. But, I'm not able currently to make that assessment myself (for either Oomph or bndtools). OTOH, I do know personally and historically the Oomph project leads and could get them involved...assuming they have time/inclination (which is a big assumption these days for any OS project).

[chrisrueger]

I think @juergen-albert mentioned to me once that they are using Oomph, maybe he could provide his perspective.

[pkriens]

I worked for two companies that used Oomph but my impression was that it require a lot of setup and maintenance work. I still have the feeling that everything we need is done quite well with a small bundle that depends on bndtools, delivered via the marketplace, and that creates the workspace. Then again, as I said, I have to see a demo how the p2/ui mechanism would work. Again, I am open for any solution but I need to see a PR and be able to play with it. Kind regards, Peter Kriens

[scottslewis]

I posted to the Oomph forum: https://github.com/orgs/eclipse-oomph/discussions/73

I still have the feeling that everything we need is done quite well with a small bundle that depends on bndtools, delivered via the >marketplace, and that creates the workspace.

Not sure why you are saying 'we'. That's not at all what I need done, nor anyone else in the bndtools user or dev communities afaict.

Then again, as I said, I have to see a demo how the p2/ui mechanism would work.

https://eclipsesource.com/blogs/2013/05/23/custom-touchpoints-in-p2/

Again, I am open for any solution but I need to see a PR and be able to play with it.

You seem to be defining/deflecting away the actual problem based upon what your needs are. The actual problem is that, as suggested by the use case above it is very difficult for new bndtools users to get to a working state...and nearly impossible to get to a working state for bndtools extensions (like remote services).

In most open projects that would mean that the bndtools dev community would look for a way to address this problem (e.g. p2, oomph, something custom), rather than asking for something so that you can 'play with it'.

[pkriens]

The problem is that I really want to help you but I just am unable to understand what you want. I'd be more than willing to merge a PR that does all what you want as long as it does not interfere badly with other parts.

I seem to be needed to do something that it is just not my current itch even though I see the value. In these cases normally someone that has the itch takes the lead and I merge their PRs after review.

Again, I really want to assist you but I won't be the lead here.

[pkriens]

https://bnd.discourse.group/t/workspace-templates/425/11

[juergen-albert]

@scottslewis Just to recap what I understood. You want to have some p2 touchpoints in bndtools, so additional steps can be triggered e.g. via Oomph? The goal being, that we can have e.g. a setup process, which installs necessary plugins, triggers the creation of a new workspace from a template and maybe also to create some projects and services via the available wizards we have!?

[scottslewis]

@scottslewis Just to recap what I understood. You want to have some p2 touchpoints in bndtools, so additional steps can be triggered e.g. via Oomph? The goal being, that we can have e.g. a setup process, which installs necessary plugins, triggers the creation of a new workspace from a template and maybe also to create some projects and services via the available wizards we have!?

@juergen-albert No. In short I don't care how it's implemented (p2 touchpoints, oomph alone or oomph with touchpoints). What I would like to have is when a feature is installed (e.g. gRPC tooling), as a result of that feature being installed a new bndtools workspace be setup...based upon an arbitrary workspace template (such as one of my remote services templates). So the desired install workflow is that the user installs a tooling feature, and ideally before Eclipse restarts the bndtools workspace is created (again based upon some workspace template...location specified as part of the feature I suppose).

I originally thought of p2 custom bndtools touchpoint(s) (e.g. a provisioningaction that when invoked downloaded a given workspace template and setup the bndtools workspace.

As pointed out by @chrisrueger, it's possible that Oomph can already do what's needed here without new p2 touchpoints. I just don't know what it's built-in capabilities/extension points are. That's why I posted a question to the oomph list; https://github.com/orgs/eclipse-oomph/discussions/73.

In short, I would like the bndtools workspace creation (based upon a given workspace template location) to be done without having the user explicitly do it themselves after the tooling install...but rather as part of the tooling install.

[peterkir]

Hi everybody,
I think that oomph is very good solution for easing the setup/bootstrap process of bndtools.
As we are part of the Eclipse ecosystem, it is viable solution.
I have been using oomph with bndtools configurations for several customers during the past years.
Tried to offer this to the bndtools contributors, but did got positive feedback.

So here are my 2 cents on this topic...
I offer to provide Eclipse oomph product and project configurations, which contain the settings for the different git projects of bndtools and related projects. Talked about his already with @juergen-albert .
And also commented on https://bnd.discourse.group/t/workspace-templates/425/9

An example of this is available at https://cdn.klib.io/oomph/index.html
There is a small collection of 1-click setup configurations (pre-requisite is a Eclipse Installer with URL handler).
If not available some more steps to download and integrate the config is required.

1. bndtools github project for contributions/debugging
2. setup of an empty eclipse workspace with bndtools
3. setup of an empty workspace containing the ecf workspace
4. setup of the cdn project itself, containing p2 repos and oomph setups (should be part of the project repos)

I volunteer to create and maintain the oomph setups for bndtools, ecf and other interested setups.
As I do this for myself and several customers since years, it is not so much additional effort.

If there is interest in this, I would appreciate a conf call to talk about this topics and define the proceeding.
The work will be done by me, but I would like to agree on the goals and proceeding.

kind regards,
the other Peter

[pkriens]

Having an Oomph setup would be great!

We can integrate it in bndtools.org, add it to the installation details. This site is maintained in https://github.com/bndtools/bndtools.github.io

If there is some code that can be used to make custom installers, I'd prefer to have that integrated in the bnd repo to simplify maintenance.

can you make (a) PR(s)?

[peterkir]

yes I can ;-)
There a multiple aspects which can be considered.

1. bnd ecosystem providing the IDE setup for contributing (there is not yet a IDE section on CONTRIBUTING.md) - should I create a PR for it - the oomph project configuration should be part of the bnd repo - it is one file - any proposals on the location - I would also propose a direct setup link on the README like https://github.com/klibio/bndtools ?
2. bnd ecosystem providing oomph project/workspaces containing pre-configured workspace templates like ecf, gecko, ...?
3. bnd and eclipse ecosystem additional project(s) for the Projects entry set of Eclipse Installer as PR to oomph
   
   Having only project bndtools/bnd or a subfolder bnd with all relevant projects from bndtools.org on github /bnd, bndtools.workspace, bndtools.workspace.min, bndtools.workspace)
4. eclipse ecosystem providing IDE product with pre-installed bndtools e.g. epp package Eclipse IDE for OSGi development
   

[pkriens]

Just do what you think it is best, make a PR so we can play with and comment. It is tricky to oversee implications without having my hands on it.

[scottslewis]

@peterkir and I have been corresponding and examining the oomph setups that he has created. A number of his setups install bndtools.core, bndtools.pde, and bndtools.m2e features alongside a new version of Eclipse (i.e. currently set to 03-2024). These work fine.

However, what the oomph setups require is a stable (i.e. durable/not changing) url pointing to the p2 repository containing the bndtools feature(s) ...for a specific released version of bndtools....e.g. 7.0.0 and 7.1.0 (hopefully soon).

@peterkir has previsously asked about getting a version-specific bndtools repository in this discussion thread:

https://bnd.discourse.group/t/specific-release-url-for-each-release-humble-request/289

It appears (to me) from this discussion that the bndtools build can be modified to produce a repo for each released version, but as far as @peterkir or I know, the releng changes to satisfy this request this haven't actually been made....as the public repos still are basically 'latest' (at 7.0.0 currently, I think).

Just do what you think it is best, make a PR so we can play with and comment. It is tricky to oversee implications without having my hands on it.

Peter could do this, but the url in his setups points to a copy of the bndtools repo (7.0.0) hosted on an internal server. If the above request can be accommodated, we will update the oomph setup urls to point to the bndtools repo and create a pr.

[pkriens]

I've created a p2 exporter. There is a test case but I've not yet had the courage to activate it. Although there is a proper test case for a real build that closely models what needs to be done in our own build. Most of the work is setting up a Github action .... You can find the project task here: https://github.com/orgs/bndtools/projects/1?pane=issue&itemId=51642824

[pkriens]

Its just time ... and so little help.

[scottslewis]

Its just time ... and so little help.

Unfortunately I know nothing about the bndtools build/releng process so cannot offer direct help.

Is there an issue associated with the request? https://bnd.discourse.group/t/specific-release-url-for-each-release-humble-request/289

[scottslewis]

I've created new issues for contributing oomph setups #6083 and creating p2 repos for specific versions of bndtools for those oomph setups #6084

[pkriens]

Can we now work on closing this since we have a constant URL for the p2 repo?