including newrelic buildpack

[kiranpatel11]

Describe the Enhancement

Can the newrelic buildpack be included in this java builder.

Motivation

This java builder seems to be including other application monitoring platforms (datadog, google stakedriver, azure application-insights), Including newrelic buildpack would make it useful for those who use the platform.

[dmikusa]

Long term yes. Short term, there is a problem preventing this.

The Paketo builders have a lot of layers to them, so many that we are getting close to the hard limit supported by Docker. We're not adding them now, because of this hard limit. There is some work that needs to be done to squash down the layers on the builder and when that is done we will add in all of the APM related buildpacks.

[kiranpatel11]

In meanwhile, whats your recommendations....

Shall we create our own custom builder with buildpacks that we need ?

[dmikusa]

The documented process right now is a custom build command.

pack build samples/java --volume "$(pwd)/binding:/platform/bindings/skywalking" -b urn:cnb:builder:paketo-buildpacks/java -b paketo-buildpacks/apache-skywalking

By putting the Paketo Java buildpack first, then the APM buildpack you want to use (you can substitute any of them for skywalking in the example above) it let's the normal build happen and then the APM buildpack runs last, which works fine for the APM buildpacks.

https://paketo.io/docs/howto/app-monitor/#apm-buildpacks-with-paketo-builders

[kiranpatel11]

on the other note, do you know how to achieve the same without pack but using the /cnb/lifecycle/creator.

probably the creators -buildpacks arg can be used but it accepts directory only, which is a bit cumbersome.

[dmikusa]

I don't sorry, I'm not that familiar with using creator directly. You could try asking on the CNB Github or Slack.

[kiranpatel11]

Long term yes. Short term, there is a problem preventing this.

The Paketo builders have a lot of layers to them, so many that we are getting close to the hard limit supported by Docker. We're not adding them now, because of this hard limit. There is some work that needs to be done to squash down the layers on the builder and when that is done we will add in all of the APM related buildpacks.

Just a question on builder images - why does the builder image so large and have these many # of layers? I am not an expert on the buildpacks, but my understanding is that the builder contains the buildpacks definitions(textual metadata only), which if detected to be used, the buildpack would download the required binaries/packages to contribute the layer needed by that particular buildpack. for example, if my application is a springboot app, by using the base builder, would it download the binaries required to build Node.js, .Net Core, Go, etc ? which are never going to be used.

[dmikusa]

Long term yes. Short term, there is a problem preventing this.

The Paketo builders have a lot of layers to them, so many that we are getting close to the hard limit supported by Docker. We're not adding them now, because of this hard limit. There is some work that needs to be done to squash down the layers on the builder and when that is done we will add in all of the APM related buildpacks.

Just a question on builder images - why does the builder image so large and have these many # of layers? I am not an expert on the buildpacks, but my understanding is that the builder contains the buildpacks definitions(textual metadata only), which if detected to be used, the buildpack would download the required binaries/packages to contribute the layer needed by that particular buildpack. for example, if my application is a springboot app, by using the base builder, would it download the binaries required to build Node.js, .Net Core, Go, etc ? which are never going to be used.

The builder image is larger because of a couple of reasons:

1. It contains the base image for build time. This is often larger because we include more tools and dev libraries/source code that are often only required at build time.
2. The builder contains a copy of all of the buildpacks that are available in that builder. Each individual buildpack is not that large. A couple of Go static binaries and some text data (it does not include buildpack installed dependencies like a JVM or Go SDK). The challenge is that buildpacks are intentionally kept small and composable, so that like puzzle pieces you can arrange them together for different use cases. This unfortunately results in an explosion of layers on the builder.

The buildpack tooling can certainly do better about this, but at the moment it basically just pushes all of the buildpacks into the image each on its own layer.

[rrjohnson85]

The Paketo builders have a lot of layers to them, so many that we are getting close to the hard limit supported by Docker. We're not adding them now, because of this hard limit. There is some work that needs to be done to squash down the layers on the builder and when that is done we will add in all of the APM related buildpacks.

For what it's worth, this is now a problem with v13.0.0 and later when using New Relic. Until this is addressed, does it make sense to remove another buildpack(s) (e.g. datadog, node, yarn) and let users add those that they might need? I realize this is a breaking change, but New Relic customers cannot upgrade otherwise from what I can tell.

[dmikusa]

We can't easily remove buildpacks because folks now expect them to be there and it's part of the contract with what we ship.

You can work around this by using the buildpack-less builder though.

pack build samples/java -B paketobuildpacks/builder-jammy-buildpackless-tiny -b paketobuildpacks/java -b paketo-buildpacks/new-relic

or

tasks.withType<BootBuildImage> {
    builder.set("paketobuildpacks/builder-jammy-buildpackless-tiny")
    buildpacks.set(listOf("paketobuildpacks/java:13.0.1", "paketobuildpacks/new-relic:8.13.0))
}

However, we've seen this to be more of an issue with Spring Boot Build Tools, and not pack. See #1392 (comment)

[rrjohnson85]

Thanks @dmikusa, that worked and wasn't aware of the buildpackless builders.