2.11.16

Features

Support GCP Secret Manager

This release adds support for using GCP Secret manager to supply secret values for steps environment.

In order to enable the secret manager, you should specify the type by setting METAFLOW_DEFAULT_SECRETS_BACKEND_TYPE to gcp-secret-manager or specifying it in the decorator

@secrets(sources=[{"type": "gcp-secret-manager", "id": "some-secret-key"}])

METAFLOW_GCP_SECRET_MANAGER_PREFIX can be set in order to not have to write full secret locations.

Support Azure Key Vault

This release also adds support for Azure Key Vault as a secrets backend. Specify az-key-vault as the secret backend type to use this.

Same as with the other secret managers, we provide a prefix config to avoid having to repeat common parts in the secret keys. Configure this by setting METAFLOW_AZURE_KEY_VAULT_PREFIX

Note: Currently only Secret object types are supported when using Azure Key Vault.

@parallel for Kubernetes

This release adds support for @parallel when flows are run --with kubernetes

Example:

@step
def start(self):
    self.next(self.parallel_step, num_parallel=3)

@kubernetes(cpu=1, memory=512)
@parallel
@step
def parallel_step(self):
...

Configurable runtime limits

It is now possible to configure the default timeout for the @timeout decorator. This can be done by setting METAFLOW_DEFAULT_RUNTIME_LIMIT in the environment, or in a config.json

Improvements

Resumed flows should record task competions correctly

Fixes an issue where tasks that were cloned from a previous run by resume would not show up as completed on the Metaflow UI due to missing metadata

Fix accessing task index of a foreach task

There was an issue accessing the index of a foreach task via the client. With this release it is possible to do the following

from metaflow import Task
task = Task("ForeachFlow/123/foreach_step/task-00000000")
task.index

What's Changed

• [runtime limits] make runtime limits configurable by @valayDave in #1834
• [Ready for Review] fix bug where client can not access foreach stack by @darinyu in #1766
• [Ready for Review] add attempt_ok flag so that UI will not show up unknown node by @darinyu in #1830
• pluggable azure credentials provider by @oavdeev in #1756
• Secret Backend Support for Azure Key Vault by @iamsgarg-ob in #1839
• reducing the dep version to 4.7.0 (#47) by @iamsgarg-ob in #1840
• [@parallel on Kubernetes] support for Jobsets by @valayDave in #1804
• Pluggable GCP auth by @oavdeev in #1841
• GCP secret manager support by @oavdeev in #1842
• [jobsets] py3.5 compatibility fixes. by @valayDave in #1844
• Support Python 3.5 for tests by @savingoyal in #1843
• py3.5 compatibility fixes [azure/gcp/jobsets] by @valayDave in #1848
• [version bump] for release by @valayDave in #1847
• [OB-625] adding metaflow/cron annotation to argo workflows by @iamsgarg-ob in #1852

New Contributors

• @iamsgarg-ob made their first contribution in #1839

Full Changelog: 2.11.15...2.11.16