Importable packages

[HadrienPatte]

Currently all the go packages that power chisel are under an internal directory, making them non importable from other go projects. The goal of this issue is to start a conversation and gauge interest in making some or parts of those packages importable by moving them out of that internal directory.

In particular, it would be very useful for types and functions related to manifest files to be importable from other projects in order to avoid duplicated code and rely on a single source of truth implementation for parsing those manifest files. These manifest files parsing functions would be very useful to add support for scanning chisel manifests in SBOM-generating tools like trivy.

Symbols that I believe would be valuable to be importable outside of this repository:

Most of the jsonwall package:

• jsonwall.DB
• jsonwall.ReadDB()
• jsonwall.DB.Get()
• jsonwall.DB.Iterate()
• jsonwall.DB.IteratePrefix()
• jsonwall.Iterator
• jsonwall.Iterator.Next()
• jsonwall.Iterator.Get()

Some of the manifest package:

• manifest.Package
• manifest.Slice
• manifest.Path
• manifest.Content
• manifest.Manifest
• manifest.Read()
• manifest.Manifest.IteratePaths()
• manifest.Manifest.IteratePackages()
• manifest.Manifest.IterateSlices()
• manifest.Manifest.IterateContents()
• manifest.Validate()

Overall, the manifest package would be nice to have public but the jsonwall is the minimum required to conformably manipulate chisel manifests. One advantage of the jsonwall package is that it currently has no dependency on any other internal package.

[cjdcordeiro]

Hi @HadrienPatte

sorry for the late reply.

First of all, thank you for the amazing work at aquasecurity/trivy@main...HadrienPatte:trivy:chisel! This is how we see the adoption of Chisel manifests happening upstream and I'm thrilled to see your take on it.

What you are asking in this issue is definitely the right move 👍 (especially for jsonwall).

We will be looking into this over the next few weeks (also considering the holiday break), as there are a couple of things we need to revisit:

1. we want to double-check that there are no reserved symbols and that the pkgs can indeed be moved out of "internal", and
2. since Chisel is currently AGPL, we need to figure out if the best way is to move these pkgs to a different project, or to add a pkg-specific license within this project.

(FYI @letFunny @rebornplusplus @zhijie-yang )

[letFunny]

I think we could expose jsonwall and manifest reading depending on what we decide and your questions above. Exposing manifest writing and the convenience functions we use internally would be quite a bit more tricky because of the dependencies with internal packages.

EDIT: Essentially the symbol list above would not require us to export any internal dependencies.

[letFunny]

Hi @HadrienPatte, first of all thank you for your contributions and for raising this issue with the comprehensive list of functions that you need. We have started working on your suggestion at #182 (as a draft PR for now). There are still some bureaucracy questions that we need to answer before merging the code, for example regarding licensing and, as @cjdcordeiro said, we are now going into the Christmas break, but we'll continue to work on the PR once we are back.