update docs for Getting Started with Nextflow Plugins

[stevekm]

The current documentation page for Nextflow Plugins has a ton of great information on how plugin source code should look and works

https://www.nextflow.io/docs/latest/plugins.html

However, in practice, I am having a really hard time getting started with actually creating a Nextflow Plugin;
nextflow-io/nf-hello#18
nextflow-io/nf-prov#25

I suspect this is due to an overall lack of familiarity with the Java / Groovy ecosystem and the Gradle build process.

Over the past years, the Nextflow Scripts page has been heavily updated to include links to relevant Java and Groovy official docs; perhaps we could get similar treatment for the Nextflow Plugins?

For example, I found the official Gradle docs here;

https://docs.gradle.org/current/userguide/getting_started_eng.html

However they are quite large, and cover a lot of different topics, and its really hard to tell which topics are most relevant for Nextflow Plugin creation, especially because its not immediately clear how the terminology used in those docs matches to what we are doing when creating a Nextflow Plugin.

It would be extremely helpful if some of the Nextflow Groovy & Java experts could help to give more pointed guidance to help us non-Java programmers :) figure out where to get started in gaining the background familiarity needed to be able to navigate Nextflow Plugin building.

An example; the Nextflow Getting Started page describes the usage of "SDK Man"; had I not happened to read this page in the past, I would not have known about this Java manager. Things like "installing the correct version of Java and Gradle" were more difficult and confusing when using the more familiar conda, switching to SDK Man helped a lot.

Other things that have proven confusing, is that the "default Gradle project init" process results in files that don't quite match whats included in the example nf-hello and nf-prov repos, so its been difficult to determine which elements from those repos are standard Gradle components or are customized for these repos and Nextflow usages, and how to best replicate those functionalities in a fresh new custom plugin repo.

Perhaps we can get these, and other insights, added to the docs as well. Thanks :)

[stevekm]

same issue here edn-es/nf-plugin-template#5

despite all the docs and examples, every example of Nextflow Plugins so far seem to break when you try to build them.

Is there some kind of "environment.yml" available for these build processes that can be used to ensure that all the correct versions of the required development build dependencies are available and used for building? At the very least, seems like docs are required to tell how to build Nextflow Plugins successfully

[bentsherman]

To address your immediate problem, most plugins are sensitive to the Nextflow version that you build with. So you can't always build a plugin against the latest Nextflow version, especially if that plugin hasn't been updated recently. This is because the Gradle configs need to be kept in sync to some extent or else things begin to break in weird ways.

it would be nice to automate this update process with a plugin template, but failing that, we should at least document which parts need to be kept in sync with Nextflow

[stevekm]

I have made more progress and have started accumulating some notes here in this demo prototype Nextflow Plugin repo; https://github.com/stevekm/nf-niceutils/blob/main/NOTES.md

based on the template repo here https://github.com/edn-es/nf-plugin-template

might be useful for this, or alternatively, perhaps that template repo and its web page https://edn-es.github.io/nf-plugin-template/ might be good places for expanded docs for Nextflow Plugins (to be eventually pulled into something like the Nextflow Training web pages)