Skip to content

Contribute community partner specific test summaries

Jump to bottom
Wei Ouyang edited this page Jul 24, 2024 · 10 revisions

Warning: The following content is outdated

Setup

A partner may generate test summaries.

In the partner section of collection_rdf_template.yaml we need to specify your partner_id (e.g. 'ilastik') and your workflow which creates and deploys your test summaries.

The partner's test workflow

Dedicated partner test workflows are triggered by the collection CI trigger_partner_wf.yaml for any newly added resource version. For this to work the partner repository needs to invite @bioimageiobot as collaborator and the partner repository needs to implement a GitHub Actions workflow which is callable from another workflow and accepts a pending_matrix input:

on:
  workflow_dispatch:
    inputs:
      pending_matrix:
        description: 'json encoded gh matrix of pending validations for new or updated resources (default: test on all resources)'
        required: true
        default: '{"include": [{"resource_id": "**", "version_id": "**"}]}'
        type: string

If the partner's test itself changes due to a new release, etc, or another test is added, it is the partner's (CI's) responsibility to trigger this workflow on the appropriate resources (or all).

The workflow should generate test summaries for the resource versions given by pending_matrix. The RDFs of any resource version can be found in the gh-pages branch of the collection-bioimage-io repository as https://github.com/bioimage-io/collection-bioimage-io/tree/gh-pages/rdfs/<resource_id>/<version_id>/rdf.yaml

Test summaries are mappings with the following keys saved as .yaml files:

bioimageio_spec_version: # str
bioimageio_core_version: # str
error: # Union[None, str, Dict[str, Any]]
name: # str
nested_errors: # Optional[Dict[str, dict]]
source_name: # str
status: # Union[Literal["passed", "failed"], str]
traceback: # Optional[List[str]]
warnings: # dict

The current keys are defined here.

The partner's GitHub Actions workflow should generate such test summaries and deploy them as configured in https://github.com/bioimage-io/collection-bioimage-io/blob/main/collection_rdf_template.yaml :config:partner:<partner entry>:test_summaries as individual yaml files test_summary*.yaml in the subfolder <resource_id>/<version_id> (example).

Integrating partner test summaries

The collection-bioimage-io CI combines all partner and the bioimageio core test summaries and deploys this test summary to https://github.com/bioimage-io/collection-bioimage-io/blob/gh-pages/rdfs/<resource_id>/<version_id>.test_summary.yaml, e.g. here from which the bioimage.io website renders them.

Examples overview

ilastik

deepimagej:

Clone this wiki locally