Unified plugin architecture

[hdonnay]

Design for a unified plugin architecture

Problem

The initialization and configuration of the modular components of claircore have been evolved over time. The original interfaces were designed to allow for encapsulation, but various demands from inside and outside claircore have led to a rickety and inefficient tower of abstractions. As an example, I'll briefly walk through the "indexer" type.¹

The libindex.Libindex constructor currently takes a slice of *indexer.Ecosystem values to which the whiteout ecosystem is unconditionally added. These are used to create a slice of de-duplicated scanner interfaces of the various known types. These are then type-erased into VersionedScanners. These scanners are then noted in the database and used to calculate the "state" value of the indexer. A LayerScanner is then constructed, which does the Ecosystem process again and then runs type assertions on the returned types to see if they implement Configurable interfaces and calling them appropriately, not returning errors if they fail.

When a request is handled via the (*Libindex).Index method, the ControllerFactory function (of which indexer/controller.New is the only known implementation) calls the LayerScanner which takes the (probably) constructed scanners and calls them with provided *claircore.Layer values, type-erasing and then type-asserting multiple times along the way.

The Matcher flow is substantially similar, and the Updater flow is more complicated due to the need for multiple rounds of configuration.

Requirements

An ideal solution would:

1. Prevent extraneous object construction.
2. Allow for validating passed configuration reporting a configuration failure.
3. Allow for both in-tree and out-of-tree components to be configured uniformly.
4. Allow for pooling of constructed objects.
5. Allow for uniform documentation of components and their configuration.

Solution

Conditions

The proposed solution has 3 parts:

1. Uniform Naming
2. Uniform Registry
3. Uniform Configuration Validation

Uniform Naming

Adopting a uniform naming scheme would allow all code working with pluggable components to operate on IDs instead of needing instantiated objects to determine their names. This would also allow for future use cases where processes manipulate the IDs in the system without needing to import Go packages.

URNs are proposed for this purpose. They are human-readable, have a well-documented encoding, and can be parameterized to allow for experimentation and expansion. By adding additional semantics into the namespace-specific part of a URN, we can easily ensure the validity of IDs passed through the system.

Uniform Registry

A uniform registry system would allow for plugins for any component of the system to make their ID known and meaningful within the system without needing to modify Claircore code. This makes out-of-tree development of components much easier and makes Claircore's creation of plugins much more uniform.

A new Go package is proposed for this purpose. It should be in a low-dependency module outside of the claircore module to make sure any diamond problems are avoided.

Uniform Configuration Validation

Uniform configuration validation would solve the confusion of when configuration is applied and to what components. The passed configuration can be checked outside of the plugin, obviating the need for all plugins to implement some validation and error reporting (or more likely, none).

JSON Schema is proposed for this purpose. While JSON Schema is not an IETF RFC yet, it is well-defined and several high-quality and highly interoperable implementations exist. The format allows for descriptions at many points in the data structure, making it possible to display actionable help when validation fails.

Concerns

Using JSON Schema to describe valid configurations does mean that all valid configurations must be able to transit JSON cleanly. This is not a requirement currently. This may cause issues in the future if a configuration is desired to be sufficiently complex.

This concern is mitigated somewhat by the fact that the primary user of Claircore, Clair, has recently made changes to the configuration format it accepts such that it is guaranteed to be able to be serialized to JSON.

Implementation

An RFC 8142-compliant URN implementation will be created in the toolkit module. It will have additional helpers for Claircore-specific names.

A generic registry will be created in the toolkit module. The proposed interface is:

func All[T any]() []string
    All reports the names of the plugins that are registered for the given type
    parameter.

func Default[T any]() []string
    Default reports the names of the plugins that are default-enabled for the
    given type parameter.

func Register[T any](name string, desc Description[T]) error
    Register registers the provided description with the provided name in the
    type-specific registry indicated by the type parameter.

    Register may report errors if the name is already in use, or if the provided
    name is not valid.

type Description[T any] struct {
	// JSON Schema to validate a configuration against.
	// See https://json-schema.org/ for information on the format.
	ConfigSchema string
	// New is a constructor for the given type.
	//
	// The passed function will unmarshal a configuration into the provided
	// value. JSON is the default format, unless a Capability flag indicates
	// otherwise.
	New func(context.Context, func(any) error) (T, error)
	// Capabilities flags.
	//
	// Meanings are set per-type.
	Capabilities uint
	// Default signals that the plugin should be enabled by default.
	Default bool
}
    Description is a description of all the information and hooks to construct a
    plugin of type T.

func GetDescription[T any](names ...string) ([]Description[T], error)
    GetDescription returns Descriptions identified by the names in the registry
    indicated by the type parameter.

    An error will be reported if an unknown name is provided or if the type
    parameter has no names registered for it.

    The returned slice of Descriptions will have the same arity and order
    as the input names. That is, given `var names []string` and `var out
    []Description`, the i-th string is the i-th Description and vice-versa.

Implementations will be expected to register in an init function, such as:

package foo

import (
	"context"

	"github.com/quay/claircore/toolkit/registry"
	"github.com/quay/claircore/indexer"
)

func init() {
	if err := registry.Register[indexer.Scanner](name, desc); err != nil {
		panic(err)
	}
}

const name = `urn:claircore:indexer:packagescanner:example?=version=1`

var desc = registry.Description[indexer.PackageScanner]{
	ConfigSchema: embeddedSchema,
	New: func(ctx context.Context, _ func(_ any) error) (indexer.PackageScanner, error) {
		return New(ctx)
	},
}

//go:embed configschema.json
var embeddedSchema string

type localScanner struct{}
func New(ctx context.Context) (*localScanner, error) {}

Claircore will maintain registration hooks for in-tree plugins via tooling.

The interfaces to libindex.New and libvuln.New will be changed appropriately to work in names instead of VersionedScanner instances.

Concerns

Small thunks will be needed to adapt current code, which may bloat implementations while a conversion is in-flight.

Configuration keys may require adapting -- a temporary function to guess at renames along with documentation should allow users to migrate.

Footnotes

1. "Indexers" were originally called "Scanners" and that name is still used in the code. This is more confusing when one considers that the component that runs indexers is call the "indexer". This document will attempt to use "indexer" consistently, but may use "detector" for clarity. Code snippets may contain "Scan". ↩