build failure with useArtifactCachingProxy=true and dependency with version range

[sephiroth-j]

Reproduction steps

1. add a dependency with a version range. for example

        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty</artifactId>
            <version>[1.2.0,1.2.99)</version>
            <scope>test</scope>
        </dependency>

2. perform a build with buildPlugin and useArtifactCachingProxy enabled

buildPlugin(
  jdkVersions: [17],
  platforms: ['linux'],
  useArtifactCachingProxy: true,
)

Expected Results

no build failure

Actual Results

error claiming that no versions are available for the dependency with the version range.

21:01:48  [ERROR] Failed to execute goal on project dependency-track: Could not collect dependencies for project org.jenkins-ci.plugins:dependency-track:hpi:6.0.0-SNAPSHOT
21:01:48  [ERROR] No versions available for io.projectreactor.netty:reactor-netty:jar:[1.2.0,1.2.99) within specified range
21:01:48  [ERROR] -> [Help 1]
21:01:48  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project dependency-track: Could not collect dependencies for project org.jenkins-ci.plugins:dependency-track:hpi:6.0.0-SNAPSHOT
21:01:48  No versions available for io.projectreactor.netty:reactor-netty:jar:[1.2.0,1.2.99) within specified range

Anything else?

below are logs of two examples, one with useArtifactCachingProxy=false and one with useArtifactCachingProxy=true

useArtifactCachingProxy=false
build for jenkinsci/dependency-track-plugin@f8e2281
jenkins run
logfile

useArtifactCachingProxy=true
build for jenkinsci/dependency-track-plugin@9e190d2
jenkins run
logfile

[timja]

(not sure on why its failing)

Why are you using a version range? Can't you just use a fixed version and then use a dependency update to keep it up to date?

[dduportal]

Hello @sephiroth-j , +1 with @timja : what is the rationale behind using a version range?

It does not seem, at first sight, an acceptable use case for ACP (I might be mistaken but I need insights).

[sephiroth-j]

I don't know why you are questioning the use of version ranges. This is a normal function of Maven. The key point is this: useArtifactCachingProxy is active by default, so builds can break out of the blue and it is not clear why!

If ACP can't handle this, then it should be clearly communicated, there should at least be a warning when using version ranges with ACP enabled and maybe it should not be enabled by default at all.

p.s.
And it has even worked in the past with implicitly active ACP and version ranges.
https://ci.jenkins.io/job/Plugins/job/dependency-track-plugin/job/master/284/console
jenkinsci/dependency-track-plugin@5fa40e3

But something apparently broke at the beginning of October.
https://ci.jenkins.io/job/Plugins/job/dependency-track-plugin/job/master/285/console
jenkinsci/dependency-track-plugin@bf85e64

[dduportal]

I don't know why you are questioning the use of version ranges. This is a normal function of Maven. The key point is this: useArtifactCachingProxy is active by default, so builds can break out of the blue and it is not clear why!

If ACP can't handle this, then it should be clearly communicated, there should at least be a warning when using version ranges with ACP enabled and maybe it should not be enabled by default at all.

p.s. And it has even worked in the past with implicitly active ACP and version ranges. https://ci.jenkins.io/job/Plugins/job/dependency-track-plugin/job/master/284/console jenkinsci/dependency-track-plugin@5fa40e3

But something apparently broke at the beginning of October. https://ci.jenkins.io/job/Plugins/job/dependency-track-plugin/job/master/285/console jenkinsci/dependency-track-plugin@bf85e64

I'm not saying version should or should not be used. But I'm asking about the rationale about using version range: pinning dependencies is a pattern helping to have reproducible builds and that is known to work over hundreds of other plugins.
It's the first time I see it on a Jenkins plugin project, hence the questioning: there is a reason for it to be used.

ACP has been introduced for sustainability reasons, including decreasing the amount of bandwidth consumed by the Jenkins project from the artifactory instances repo.jenkins-ci.org.
It has been requested by JFrog as a mandatory point to continue the sponsorship of repo.jenkins-ci.org. It helped to avoid consuming around 10 Tb per month.

It also helps decreasing the cost of the Jenkins infrastructure to keep it alive (as it is a sponsored-only infrastructure).

I understand your frustration about builds failing which most probably slows you down or blocks you.
But it is the first time I see version ranges used on a Jenkins plugin so I would like to understand. ACP has been built with trades-off (maintenance effort, sustainability, features, stability) depending on the context. We did not take in account the version pattern so it has never been tested despite being a Maven feature.
If we want to understand the issue and fix ACP, we need to understand the context in order to provide a solution.

[dduportal]

We discussed this topic during today's SIG Infra weekly meeting:

• This is legit request from the "Jenkins Infra. support" view as ACP should not fail for such case (Maven emits an HTTP request which should be proxied: it makes no sens to me why it is failing)

  • Note: the Jenkins Infra team consider this issue worth diagnosing, but it is clearly not on top of our priority list, so we do not really have an ETA. We keep it in our milestones: if we carries it over more than 4 weeks, then it will go to the backlog as it would mean we simply not have the bandwidth.

• On the Jenkins Contributor side, using version range is something we want to discourage: it is an obstacle to reproducible builds and proper dependency management. However, we should document the expectation in our developer documentation (ping @MarkEWaite @kmartens27 @gounthar @darinpope)

  • The expected pattern for dependencies is to have pinned dependencies updated by dependabot or renovabot

Of course I could miss a use case or a solid reason to use version range in this context: if it is the case, please share your knowledge so we can build on top of it

[sephiroth-j]

I use version ranges only for dependencies in the test scope. I am doing this to reduce the noise of pull requests from the bots and the effort of reviewing and merging them. I consider the affected dependencies very stable and they are using semantic versioning. I do not require a particular patch version. And because I would use the latest patch version anyways, I'm using the version range feature of Maven. Though this will not meet the requirements of a reproducible builds in whole, it still does in the part that actually counts and is the most important: the shippable artifact, the plugin itself.

I also personally believe, that no one should develop a plugin for a software for a specific patch version of this software, unless there is a really good reason do to so. This holds true for Jenkins. We are developing a plugin and define a minimum required version of Jenkins. This version may be the latest stable at this time. But some time later, a new stable version of Jenkins gets released. This may be just a patch or new minor or even major version. In such case, the plugin version very often stays the same and only Jenkins is updated. But what now? The plugin is used with a version of Jenkins that it has not been tested with. In most cases this is fine because it will continue to work. But it could also be the case that it does not. Version ranges can come in handy in this situation because they allow me to define the range of supported versions. Once a new minor or major version comes out, I have to adjust this range and test the software against this new version. And for Jenkins, I would like to build/test against this new version but without increasing the minimum required version when there is no reason to do so.

[dduportal]

Thanks @sephiroth-j for the explanations. Despite having a different opinion than yours, I believe you feedback is legitimate and as such we'll check why ACP did not behave as expected.