2022 Roadmap

[dmikusa]

It's that time of year! The Paketo Project is looking for your feedback to help plan the project direction for 2022.

Please take a moment and comment/upvote below. Tell us what you think Paketo Buildpacks should focus on in 2022?

---

All feedback is encouraged and appreciated. The Paketo steering committee will use up-votes to gauge the interest level in a topic and we will group feedback by the following categories for the roadmap:

• Non-technical across the whole Paketo Project
• Technical across the whole Paketo Project
• Non-technical for a specific Paketo project sub-team
• Technical for a specific Paketo project subteam

Thanks & looking forward to hearing from everyone!

[ForestEckhardt]

Bring new SBOM layer format to as many buildpacks as possible

[ForestEckhardt]

Finish a technical overhaul and rewrite of PHP buildpack

[ryanmoran]

Reloadable process types for PHP and Ruby

[ryanmoran]

Yarn v2/v3 support

[ryanmoran]

ARM64 stacks

[ryanmoran]

Function buildpacks

[lance]

What do you mean by this?

[dmikusa]

In a nutshell, a buildpack that allows you to push a single function (basically less than a full application) and the buildpack would handle turning the function into something that's runnable (possibly by wrapping it with additional functionality).

[lance]

Right on - I am familiar with the concept (see https://github.com/knative-sandbox/kn-plugin-func). We have found that with interpreted languages, it's possible to use the standard buildpacks, because the runtime can employ a library or module to do the work of wrapping and invoking a function at startup vs during compilation (for example, https://github.com/boson-project/faas-js-runtime). For compiled languages such as Go, we have found the necessary shim to be quite small https://github.com/boson-project/packs/tree/main/buildpacks/go/faas. I think the difficult thing here will be to find across-the-board agreement on what the API should be for these.

[sophiewigmore]

Increased flexibility around stack creation

[griggheo]

Update the base stacks more often. Here we are in March 2022 and the base stack is still based on Ubuntu 18.04:

https://github.com/paketo-buildpacks/base-stack-release

[loewenstein]

I'd actually expand that to support multiple Ubuntu LTSes in parallel.

For some a rather old base OS would be seen as a problem. Also, there are some CVEs in Bionic that are marked as "won't fix, because there is no compatible fix" by Canonical.

On the other hand side, I can imagine that older LTSes will suffer fewer zero day exploits.

[agracey]

I wonder if this might be better enabled once upstream implements https://github.com/buildpacks/rfcs/blob/main/text/0096-remove-stacks-mixins.md

[loewenstein]

In terms of what OSes a buildpack supports, maybe - if some can’t support *, but would support multiple Ubuntu versions. However, this is not only about what the buildpacks support - Paketo also provides and maintains a Stack. Even if the notion of a Stack gets removed (not sure if the RFC goes that far actually), we will still need maintained base image layers, no?

[dmikusa]

Implement Java RFC 0007 and improve the story for using Java buildpacks in low-memory situations

[dmikusa]

Expand the supported set of Java application servers available through the Java buildpack. Presently, we fully support Apache Tomcat. Presently being worked developed are Liberty and Apache Tomee. If there is another application server we should support, please reply to this comment with the name.

[eddumelendez]

Wildfly
Payara Server and Micro

[dmikusa]

Expand the support for APM (application monitoring) tools through Pakto Buildpacks. If of interest, please reply with your APM tool of choice.

[garethjevans]

Dynatrace

[ThomasVitale]

OpenTelemetry

[dmikusa]

Improve the buildpack authoring story. We primarily document using the Paketo buildpacks, but we also provide buildpack authoring libraries and tools. Improve the story around authoring so it's more approachable to write a buildpack using Paketo libraries and tools.

[dmikusa]

Publish language-specific builders.

Right now we publish one big builder, with all the language families. In addition, we could publish what would be smaller language-specific builders that include only a subset of the buildpacks for a particular language.

[dsyer]

Stop downloading the JVM for every build. It’s the same binary every time.

[whazor]

These binaries should already be cached in the cache layer.

[dsyer]

I’m guessing you are not a Java developer?

[agracey]

They "should" be cached already. If they aren't in your experience, that's a bug as I understand it. Either in paketo or in the platform that's running paketo.

[dsyer]

They are only cached for a successful build and only per project. So you can have a really frustrating experience with failed builds or building multiple projects.

[dmikusa]

Yes, caching is presently scoped per application so if you build app-a and app-b that equates to two downloads of assets :( It also only saves cached assets on a successful build, so if your first build fails it can be very frustrating.

[making]

support "read-only" image

[dmikusa]

@making Can you expand on this? I want to make sure I know exactly what you mean by immutable images. Thanks

[making]

Fixed to "read-only image".
that allows docker to run with --read-only.

I think it's helpful to avoid CVEs such as CVE-2021-42550 that requires the write access to logback.xml.

Currently It doesn't work with Spring Boot as follows. This may be a framework issue (spring-projects/spring-boot#21519) rather than a buildpack issue.
It would be great if there was no blocker to run the image read-only on the Packeto side.

Also, there seems to be a workaround when using a Dockerfile, and I'm wondering if there is a way to create an immutable image out of the box with buildpacks.

$ docker run --read-only --rm docker.io/library/demo:0.0.1-SNAPSHOT
Setting Active Processor Count to 8
Calculating JVM memory based on 11536432K available memory
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx10941851K -XX:MaxMetaspaceSize=82580K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 11536432K, Thread Count: 250, Loaded Class Count: 12166, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 128 container CA certificates to JVM truststore
WARNING: Unable to add container CA certificates to JVM because /layers/paketo-buildpacks_bellsoft-liberica/jre/lib/security/cacerts is read-onlySpring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=8 -XX:MaxDirectMemorySize=10M -Xmx10941851K -XX:MaxMetaspaceSize=82580K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.4)

2022-03-09 01:56:01.842  INFO 1 --- [           main] com.example.demo.DemoApplication         : Starting DemoApplication v0.0.1-SNAPSHOT using Java 11.0.14.1 on d9bae77f71e1 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2022-03-09 01:56:01.846  INFO 1 --- [           main] com.example.demo.DemoApplication         : No active profile set, falling back to 1 default profile: "default"
2022-03-09 01:56:02.889  WARN 1 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to create tempDir. java.io.tmpdir is set to /tmp
2022-03-09 01:56:02.900  INFO 1 --- [           main] ConditionEvaluationReportLoggingListener : 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2022-03-09 01:56:02.934 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to create tempDir. java.io.tmpdir is set to /tmp
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:163) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577) ~[spring-context-5.3.16.jar:5.3.16]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:740) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:415) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301) ~[spring-boot-2.6.4.jar:2.6.4]
	at com.example.demo.DemoApplication.main(DemoApplication.java:10) ~[classes/:0.0.1-SNAPSHOT]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[workspace/:na]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[workspace/:na]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[workspace/:na]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[workspace/:na]
Caused by: org.springframework.boot.web.server.WebServerException: Unable to create tempDir. java.io.tmpdir is set to /tmp
	at org.springframework.boot.web.server.AbstractConfigurableWebServerFactory.createTempDir(AbstractConfigurableWebServerFactory.java:195) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:195) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:182) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:160) ~[spring-boot-2.6.4.jar:2.6.4]
	... 16 common frames omitted
Caused by: java.nio.file.FileSystemException: /tmp/tomcat.8080.6571514823119891072: Read-only file system
	at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[na:na]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[na:na]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[na:na]
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source) ~[na:na]
	at java.base/java.nio.file.Files.createDirectory(Unknown Source) ~[na:na]
	at java.base/java.nio.file.TempFileHelper.create(Unknown Source) ~[na:na]
	at java.base/java.nio.file.TempFileHelper.createTempDirectory(Unknown Source) ~[na:na]
	at java.base/java.nio.file.Files.createTempDirectory(Unknown Source) ~[na:na]
	at org.springframework.boot.web.server.AbstractConfigurableWebServerFactory.createTempDir(AbstractConfigurableWebServerFactory.java:189) ~[spring-boot-2.6.4.jar:2.6.4]
	... 19 common frames omitted

[sambhav]

This is being proposed at paketo-buildpacks/rfcs#135

[dmikusa]

Yes, that sounds like the RFC @samj1912 mentioned, which is geared towards having more secure containers. I don't know that we could get it down to a completely read-only image. Possibly for some workloads, but a lot of stuff needs a least a temp directory. In this case, Spring is creating it but I believe it's Tomcat that's using the temp directory.

Anyway, would RFC 135 suffice for your needs?

[ekcasey]

I don't know that we could get it down to a completely read-only image

I think we should have a goal of a supporting read-only root filesystem, which doesn't preclude relying on the presence of few additional writable volumes (usually a tmpfs) mounted at certain well-known locations like the temp dir.

[agracey]

Buildpack to run custom script provided by environment variable (similar to BP_NODE_RUN_SCRIPTS but language agnostic)

[dsyer]

Allow multiple profiles with different classpaths (additional jars) in JVM buildpacks. I know it's possible to write a custom buildpack but that's something of a black art and not many people will be able to benefit from it.

[dmikusa]

@dsyer Can you expand on what you're referring to with profiles? or maybe walk through a short example of what you mean? When I read this, I'm thinking of the buildpack process types that we generate (i.e. the command that runs when the image executes). Just want to confirm I'm on the same page. Thanks

[dsyer]

Process types was what I had in mind. Buildpacks already have a habit of adding jars to application classpaths (eg the service binding one). I just want to make it more flexible and give some control to the user. Examples: add actuators in prod profile; add Prometheus in metrics profile; add flyway in maintenance profile; remove web stack in batch profile.

[sambhav]

Data Science based buildpacks. Eg. - support for Jupyter, Spark, PyTorch, TF etc.

[dmikusa]

Improve the new contributor experience for Paketo projects. Essentially try to make it more approachable for someone to contribute code or docs to the project. Examples: tag issues for new contributors, provide new contributor WG, document required dev tooling/environment set up (or automate it), etc...

[agracey]

Auto-Instrumentation with OpenTelemetry

[dmikusa]

Can you expand on how you'd see this working? Would it be adding libraries that do this? Is there a particular language family where you'd like to see this?

[garethjevans]

Documentation / Guidance on how to use Occam

[sophiewigmore]

We have some GoDoc available, but not for everything. Is that helpful or would concrete examples (/how to guides) be more helpful? I also realize there's no README.

[dmikusa]

Improve dependency management

• Be more flexible with how dependencies are downloaded, i.e. support alternative repos/host rewrite
• Support pinning dependencies
• Support dependency management policy and config through builder specified configuration
• Support signature validation

[ekcasey]

Expand the set of supported stacks to include more base images. If of interest, please reply with your desired base image (e.g. ubi8, debian 9)

[ekcasey]

Better consistency across Paketo buildpacks. For example we could standardize:

1. buildpack.toml metadata schema
2. choice of language bindings (libpak vs packit)
3. testing frameworks
4. buildpack patterns and conventions more generally

[ekcasey]

Complete (and auto-generated?) reference docs

[ekcasey]

A replacement for Procfiles that makes sense with directly executed processes using the new schema.

[dmikusa]

A replacement like a totally different input file, i.e. no more Procfile, or still use Procfile as input but a difference in handling that makes it work with the new schema?

[ekcasey]

A totally different input format. The current format (a single string) made more sense historically when all processes had a single string command and were launched with bash.

To make an analogy to Dockerfiles, we currently let people do the equivalent of:

ENTRYPOINT echo hello world

But we should be allowing constructions like:

ENTRYPOINT ["echo", "hello"]
CMD ["world"]

Our current Procfile implementation has already evolved to handle creating direct processes and it can continue to evolve. We don't need to get rid of it.

But we should introduce a new schema that is geared towards a bash-less world and allows users to set both command and args.

[sambhav]

See https://github.com/samj1912/proc-descriptor-buildpack

[nrakesh]

Buildpack for R