Overview

CCD has two components - a web server for managing and orchestrating workflows which resides in a single OpenShift project, and a variable number of separate OpenShift projects which represent data contributors to the CCD system.

A User dispatches computations by submitting a workflow to chamberlain. A high-level summary of the protocol execution is detailed below.

1. User submits workflow

Conclave jobs are dispatched via chamberlain. Workflows are detailed in the Workflows section, and the submit API is detailed in the API section.

2. chamberlain generates pods for the workflow and dispatches them

For each Swift/Dataverse endpoint in the submitted workflow, chamberlain generates a single pod and launches it in the namespace corresponding to the endpoint's data owner. All the pods in a single workflow are configured by chamberlain such that they can establish TCP connections to one another and execute the computation.

3. Each pod pulls its data from designated object store

Each pod is mounted with its own data store configuration details from the namespace in which it is running. Note that these details are only visible within the namespace hosting the pod, so they are not viewable by chamberlain at any point. Each pod uses these configuration details to authenticate against Swift / DV and download its input data (along with the policy corresponding to that data) onto an emptyDir volume mounted on the pod.

4. Workflow is verified against each organization's policy engine, computation begins

The pods connect to one another, exchange policies for their data, and verify each policy against the policy engine. The policy engine itself is described in detail in the Repositories section. It the workflow is verified by every policy, then the computation begins. If, however, the workflow is rejected by any individual policy, the computation terminates and all pods disconnect.

If verified, the submitted workflow will be decomposed by conclave into a series of local and MPC computations, which will then be executed in the appropriate order.

5. When the computation is finished, output is written to a Swift volume that is accessible by chamberlain

A single party writes the output to a Swift volume which can then be retrieved by the user via the return_output API.