ADR 43: add the Artifact construction ADR #212
Conversation
|
|
||
| A new API will be introduced which instructs the Integration service to create a Tekton pipeline which will generate the | ||
| artifact. | ||
| Once built, the artifact reference will be added to the Snapshot's metadata so it can be discovered during testing or release process. |
There was a problem hiding this comment.
Why are we adding the artifact to the Snapshot? Is the intention not to replace the Snapshot CR with the produced artifact?
There was a problem hiding this comment.
For the higher-level purpose of upstreaming and decoupling services, yes, but I think that the Snapshot (or a future replacement API) should still be used by the integration service for the following:
- As a basis for generating the artifact(s) in the artifact pipelineRuns
- For running integration testing pipelines
There was a problem hiding this comment.
Ah, you are proposing to add it to the metadata, not to the spec (which is where the rest of the references are?). If that is true, I think it makes sense as a link-out to some external storage object. Is another potential option updating the status?
Some further questions assuming the above is true (likely more implementation details than anything):
- Should we support a list of references to support "rebuilds"?
- This update should fail if the CR is not present (i.e. if it has been deleted/pruned and stored in some external database). Will this prevent us from being able to rebuild these artifacts?
I agree that some integration-internal resource is likely still needed to be able to construct the artifacts and their pipelineruns. I just expecting that this would effectively require users to migrate their group-like object to no longer be Components. (we can migrate to this flow without requiring migration by auto-configuring a replacement for the "default" snapshot object)
If we are considering to remove the Snapshot as the API to communicate to a release, then we should also consider using the extractor pattern to "pass a snapshot" to integration testing pipelines as well. This will ensure that we have consistency between all tests and release processes. We will, of course, need to plan this migration from submitting a Snapshot for release to submitting some object + extractor to release.
| Since the `extractor` resource is a list, it can take multiple extractor entries if the artifact pipeline is expected to provide multiple artifacts. | ||
| Each extracted resource will be added to the Snapshot's metadata. | ||
|
|
||
| ## Consequences |
There was a problem hiding this comment.
There are many moving parts involved with this change ... and implications about other changes. Is your intention here to isolate into specific incremental proposals? I feel like we need some representation of the end-goal so that we can state how each proposal supports it (and so that we can identify cases where we might be limiting/restricting ourselves).
Some potential relationships:
- TestSubjectConstructor (related to decoupling integration, implications on build and UI)
- Remove coupling around an Application and an Application Snapshot (this will have implications for the release service)
- Potential decrease of a reliance on an Application (this will have implications on build and UI)
There was a problem hiding this comment.
Good points, I'll add them to the consequences and think on further implications.
|
|
||
| This ADR aims to cover the proposed functionality for creating additional Artifacts as part of the Konflux build/test/release process. | ||
| These artifacts would vary in type and would cover use cases where either the single built image or entire sets of images | ||
| from the user's application are needed to generate one or more artifacts. |
There was a problem hiding this comment.
Some limitations with our current approach that can be mitigated with this ADR:
- The ability to produce multiple artifacts with a single PipelineRun
- Reduced need for in-PR nudging (one of the common use cases for this is the operator bundle builds as mentioned below)
| Additionally, the integration service will wait for the pipelines to finish before starting the integration testing pipelines | ||
| in order to enable users to run tests using the new artifacts. |
There was a problem hiding this comment.
The pipeline that we are waiting for are the "artifact constructor" ones, right?
There was a problem hiding this comment.
Yes, updated to clarify.
| extractors: | ||
| - ref: 'status.results.filter(result,result.name == "IMAGE_URL")[0].value' | ||
| name: ‘metadata.labels["appstudio.openshift.io/component"]’ |
There was a problem hiding this comment.
Is this selector and extractor pattern common with the TestSubject constructor that has been previously proposed? Should it be present separately on multiple CRs or does this indicate a possible need for a common abstraction of the functionality?
There was a problem hiding this comment.
That's a good question. I think we are aiming for a similar paradigm to occur in multiple APIs, especially for the selector, but I'm not sure if a common abstraction would be required.
There was a problem hiding this comment.
Would these CRDs be managed by the same controller (i.e. integration service) or would there be separate controllers? Would this just be some pattern that could be reused within the same codebase?
| Additionally, the integration service will wait for the pipelines to finish before starting the integration testing pipelines | ||
| in order to enable users to run tests using the new artifacts. | ||
|
|
||
| ### The ArtifactConstructor API |
There was a problem hiding this comment.
Should this be a generic artifact constructor or should it be a more narrowly scoped type of API like a CollectionConstructor?
There was a problem hiding this comment.
I'm leaning towards it being a generic artifact constructor to allow for the different types of artifacts to be generated. As long as the pipelines can build something that can be referenced and fetched, I think that we should be able to support it this way.
There was a problem hiding this comment.
Are there other use cases that you have in mind?
| - name: pathInRepo | ||
| value: pipelines/bundle_generator_pipeline.yaml |
There was a problem hiding this comment.
This is a pipeline that is specific to generating operator bundles? Is one of the core tenants of the proposal not that we should abstract this further?
Basing this off of the model for building trusted artifacts, would it not be more easily extendible to specify an image which can construct and artifact (collection) based on some set of images passed to it and extract the image(s) into a list? If we construction and extraction logic into tooling, then we can potentially use a common pipeline.
At least one caveat that comes to mind:
- Build-time tests would be harder to customize on a common pipeline (are there tests that should run in it or should all tests just be run against the produced artifact in an ITS)?
There was a problem hiding this comment.
I may have erred on the side of more generic solution here - I think that we could support both directly building artifacts such as operator bundle images (or any other images), and building OCI artifacts that contain further data/metadata that can be used as described.
| kind: ArtifactConstructor | ||
| metadata: | ||
| name: example--artifact-constructor | ||
| spec: |
There was a problem hiding this comment.
In defining the constructor, we should specify a source/configuration for building the artifact. This would resemble the "normal parameters" which might exist for builds. These parameters will also be needed for the provenance (i.e. so that we can associate the artifacts produced to specific sources).
These artifacts may be desired to be built multi-arch. While a multi-arch pipeline could be selected, the specific platforms to build on would need to be selected.
There was a problem hiding this comment.
Would a PARAMS field work here? That way the users could provide their own set of parameters that could also include the source, configuration, etc.
There was a problem hiding this comment.
Something like that might work. This also relates to the conversation about PAC.
|
|
||
| ```yaml | ||
| apiVersion: appstudio.redhat.com/v2alpha1 | ||
| kind: ArtifactConstructor |
There was a problem hiding this comment.
This will result in pipelines run in users' namespaces without the use of PAC. We would lose the ability to interact with PipelineRuns as users might normally be used to (i.e. re-triggering them). How could we ensure that we still provide users with that necessary functionality?
There was a problem hiding this comment.
The integration pipelines are also run without the use of PaC, and there is already a re-triggering functionality available for them, would that be enough to cover the use cases?
There was a problem hiding this comment.
It might help in some use cases. I don't know how narrowly scoped those are.
Another use case that we need to consider is if the artifact-generating repository reference were updated, we would need to be able to test the new images/build process.
I don't know what the best way to do that is. Would it make sense to still add this Component and then designate some component's repo/configuration as the constructor argument? This feels a little wonky to me.
| The proposal here is to introduce the new ArtifactConstructor API which will define Tekton pipelines that will generate individual artifacts. | ||
| These pipelines would be executed after a Snapshot is generated and before the integration test pipelines start. | ||
|
|
||
| When each artifact generation pipelineRun is completed, the reference to the artifact is extracted and added to the Snapshot. |
There was a problem hiding this comment.
added to the same snapshot or a new snapshot will get created?
There was a problem hiding this comment.
It's added to the same Snapshot, will clarify.
| ## Use Cases in Detail | ||
|
|
||
| * A team produces operands and operators. When integration service creates a Snapshot, it uses a constructor that | ||
| actually builds a bundle. When releasing, the team releases that single bundle artifact. That constructor also has an extract |
There was a problem hiding this comment.
How this will work with the nudging functionality? today when a new operand is available a pr for updating the CSV of the bundle with the reference to it is created, and it starts the build process for the bundle.
With this proposal, the bundle build will start by the integration service without requiring a change to the bundle source in git?
There was a problem hiding this comment.
That's the general idea, the bundle builds would not require nudging unless the user explicitly wanted to have the bundle contents stored/updated in their git repository - and even then the contents could be updated as part of the release process if needed.
There was a problem hiding this comment.
Users would have to migrate to the flow proposed by the ADR. This does not remove the functionality that is present today with explicit operand/operator/bundle builds and nudging.
It does, however, reduce the demand for in-PR nudging as one of the primary drivers for that was to produce bundle images to enable testing changes.
arewm
changed the title
| @@ -0,0 +1,95 @@ | |||
| # XX. Creating Artifacts from Snapshots for testing and release | |||
There was a problem hiding this comment.
Can we change this to ADR 43? I am trying to deconflict the in-review ADRs.
Signed-off-by: dirgim <[email protected]> rh-pre-commit.version: 2.2.0 rh-pre-commit.check-secrets: ENABLED
dirgim
force-pushed
the
artifact-constructor-adr
branch
from
cf40d38 to
858ff54
Compare
| ## Context | ||
|
|
||
| This ADR aims to cover the proposed functionality for creating additional Artifacts as part of the Konflux build/test/release process. | ||
| These artifacts would vary in type and would cover use cases where either the single built image or entire sets of images |
There was a problem hiding this comment.
- Will it support non image artifacts, like rpm?
- Could you give an example of an artifact that consist of several images? Or you mean a set of images that logically create a single unit (artifact), like k8s operator + bundle?
There was a problem hiding this comment.
- Yes, it can be rpms or binaries or any other kind of artifact that can be stored somewhere and released.
- Exactly, an operator bundle is a good example, or an fbc fragment - these rely on information (mainly images) from multiple components in order to be generated.
|
|
||
| This ADR aims to cover the proposed functionality for creating additional Artifacts as part of the Konflux build/test/release process. | ||
| These artifacts would vary in type and would cover use cases where either the single built image or entire sets of images | ||
| from the user's application are needed to generate one or more artifacts. |
There was a problem hiding this comment.
Will it allow releasing a few artifacts at once? E.g. I build a few binaries in build pipeline and release them into dedicated location for each?
There was a problem hiding this comment.
Yes, I think releasing more than artifact at the same time should be supported. We are not delving too deep in the Release process here, but I would like to hear from @davidmogar or @johnbieren about their opinions on this.
There was a problem hiding this comment.
Not sure I am really following, but isn't this what we already do? We let you say where to release each container image
|
|
||
| Some limitations with our current approach that can be mitigated with this ADR: | ||
| * The ability to produce multiple artifacts with a single PipelineRun | ||
| * Reduced need for in-PR nudging (one of the common use cases for this is the operator bundle builds as mentioned below) |
There was a problem hiding this comment.
Are you aiming at replacing nudging completely or just for in-PR testing?
There was a problem hiding this comment.
I think that nudging for operator bundle builds could potentially be replaced completely, but I don't think that nudging between images in general (e.g. base image -> images that are based on it) would be replaced.
|
|
||
| * A team produces operands and operators. When integration service creates a Snapshot, it uses a constructor that | ||
| actually builds a bundle. When releasing, the team releases that single bundle artifact. That constructor also has an extract | ||
| operation to list all related images that are referenced, and that extraction generates the list for release service to operate on |
There was a problem hiding this comment.
Do I understand correctly, that all related images will be released too when releasing the bundle artifact?
There was a problem hiding this comment.
Yes, I believe that it would be possible to set up the release logic to release the bundle alongside the related images.
| * A team produces operands and operators. When integration service creates a Snapshot, it uses a constructor that | ||
| actually builds a bundle. When releasing, the team releases that single bundle artifact. That constructor also has an extract | ||
| operation to list all related images that are referenced, and that extraction generates the list for release service to operate on | ||
| * A team produces a set of images with ko. The build task generates/pushes all images with ko and then produces a "build artifact" |
There was a problem hiding this comment.
Using build task here is confusing. You didn't mean build task of the build pipeline, right?
There was a problem hiding this comment.
Good point, maybe calling it an artifact generation task would be more clear.
| The proposal here is to introduce the new ArtifactConstructor API which will define Tekton pipelines that will generate individual artifacts. | ||
| These pipelines would be executed after a Snapshot is generated and before the integration test pipelines start. | ||
|
|
||
| When each artifact generation pipelineRun is completed, the reference to the artifact is extracted and added to the original Snapshot. |
There was a problem hiding this comment.
By the artifact you mean "artifact of artifacts"?
|
|
||
| The `selector` field will be used to identify resources that should trigger the construction of new Artifacts. This will be done using CEL expressions. | ||
|
|
||
| The `resolverRef` resource will be used to define the Tekton pipeline definition which can be used to run the artifact pipelineRun. |
There was a problem hiding this comment.
Do you mean PipelineRun that creates an artifact?
|
|
||
| The `resolverRef` resource will be used to define the Tekton pipeline definition which can be used to run the artifact pipelineRun. | ||
|
|
||
| The `extractor` resource will use CEL expressions to extract the individual field values from the artifact pipelineRun. |
There was a problem hiding this comment.
Shouldn't the pipeline just define result(s) with the reference to the produced by the pipelinerun artifact?
|
|
||
| ## Consequences | ||
|
|
||
| - [integration-service] now executes additional Tekton pipelineRuns which are expected to generate artifacts which |
There was a problem hiding this comment.
I think that individual artifacts (e.g. image or bundle or ...) should be generated by the build pipeline. It does make sense to group individual (let's call "atomic") artifacts produced by build pipeline into a group and the release it as a single unit. But the group of atomic artifacts create an artifact to release which seen somewhat the same as snapshot.
| - [integration-service] waits for the artifact pipelineRuns to finish before starting the integration testing pipelines | ||
| for a given Snapshot. | ||
| - this will have implications for the [release service] as it will further remove coupling around an Application and an Application Snapshot | ||
| - Potential decrease of a reliance on an Application, which will have implications on [build-service] and the Konflux UI |
There was a problem hiding this comment.
Build Service does not rely on Application, main resource is a Component. From Build Service point of view, we can delete Application CR even today.
There was a problem hiding this comment.
Good point, I think that a better term would be the Application/Component model.
This ADR aims to cover the proposed functionality for creating additional Artifacts as part of the Konflux build/test/release process.
Signed-off-by: dirgim [email protected]