Alignment of Buildpack and Paketo libraries

[modulo11]

I would like to open up a discussion on how supporting libraries (libcnb, libpak and packit) are currently set up.

Implementing Paketo RCFs needs to be done in two libraries for the same concern, b/c they are implemented differently. Popular examples include:

• Logging
• Parsing buildpack.toml
• Handling of SBoM generation
• Dependency structures

Taking the logging example, you can clearly see differences in the output e.g. of Java family buildpacks (using libcnb and libpack) and Node.JS family buildpacks (using packit).

Another example would be the implementation of RFC-0095 which standardizes the dependency metadata. Implementation would need to be done in both libraries: libpak and packit resulting in duplicated code.

Ideas

Spoiler: I think there will be no perfect solution. Before investigating deeper into one of the ideas, reaching some sort of consensus within the community is highly anticipated.

Sacrifice either libpak or packit

Looking at the usage numbers (couple of weeks old) within Paketo, 58 buildpacks are using packit and 49 buildpacks are using libpak and libcnb. Goodby libpak.

This change would probably not come over night and put a lot of work on the Java family maintainers. Also, I'm not sure if all of the features within libpak are available in packit.

Separate Concerns

Headline might be a bit misleading, but this would be my idea: General buildpack stuff and basic implementation of the spec go into libcnb (this is the case already). Building on libcnb there would be libpak implementing the general infrastructure for all Paketo buildpacks (parsing buildpack.toml, dependency structures, logging, ...). Then there's packit providing higher-level building blocks but re-using the ones from libpak.

This is sadly not possible to do step by step as every change is potentially breaking, leading to a new major version of a library. Maybe introducing interfaces for each of the larger infrastructures (logging, parsing, dependencies etc.) with capabilities from both, libpak and packit could be a way to go. Having that in place, the community can discuss the individual approaches and later deprecate one of the implementations. Like said in the introduction, I have not yet tried if such an approach could be done in all cases.

New Utility Library

Leave libpak and packit as they are and come up with a fourth library only containing common logic like logging, parsing and dependencies. This way, libpak and packit can consume common infrastructure at their pace and new functionality can be centrally provided. Of course, this would result in releasing and bumping an additional library with the usual costs.

[dmikusa]

Sacrifice either libpak or packit

It's good to list all of the options, but I think we can throw this one out. I don't think either team would take on this level of effort, nor do I think the payoff is worth it. So while it's something we could make the decision to do at a project level, in practice I can't see it ever being implemented.

Separate Concerns

This has been the path that we've discussed briefly. Essentially, libcnb is the implementation of the buildpacks specifications. It is only an implementation of the spec, so it's not supposed to provide high-level or convenience options to authors. You can use it directly, but you may find it a little cumbersome for more involved buildpacks.

We can then move towards having libpak and packit which are both based on top of libcnb. The spec implementation will be the same so a lot of duplicate effort is avoided by doing that. It should also help ensure buildpacks are behaving and implementing the spec in the same way, although this hasn't been an issue to date.

This puts libpak and packit as "convenience" layers for buildpack authors. They can implement additional high-level features that might make implementing buildpacks easier or less work. I believe that there's room for these both to exist and for them to differ. It's a chance for teams to explore the authoring process and provide a different experience for authors. Authors can then pick the experience they like and write buildpacks in that style.

Long term, and if we still see a lot of overlap in libpak and packit, then I think what we should do is go the route of a fourth library. We can combine the two libraries into one taking the features from both and exposing both. In theory, you could do this and not have to break APIs because the new library can essentially do both. The transition for buildpack authors would jut be to switch package names.

Then as time goes on, you can look at evolving the fourth library to perhaps have a more unified experience, and since we're on a single library that's more achievable.

I think that the other possibility with a fourth library is that we put aside the current ways of implementing buildpacks and create something that's altogether new and better. If we take lessons learned from both libraries and design a wholly new and better API then there's an incentive to move to it and authors will naturally do that. The obvious risk of this is that it's not better enough, people stick with what they're doing already and we then just have yet another library floating around.

New Utility Library

I'm not opposed to this, as it makes sense to move out common functionality, but maintaining libcnb, libpak, libjvm, and libbs, I can tell you that having more libs like this is a pain. When you need libcnb to change, there's a ripple effect where you need to bump all of the other libraries and it's a lot of work. Adding another one is just going to make that worse.

There's an inherent need for libcnb and libpak to be different because they're owned by different projects, but aside from that I'd personally like to see these extra libs go away. It would reduce the pain when we have to release them.