MGMT-17267: add etcd encryption support

[mresvanis]

This PR adds OpenShift etcd encryption support for recert:

• recert now auto-discovers the seed encryption config and reads OCP encrypted resources using those keys, during recertify and ocp_postprocess
  • supports AES-GCM (openssl doesn't support AES-GCM, so not FIPS compliant anyway, we'll use Rust libs)
  • supports AES-CBC (via openssl for FIPS compliance)
• add cmd line args for target's {kube,openshift,oauth}-apiserver encryption configs to keep target's keys and encrypt all resources with those before committing to etcd
• when seed encryption is enabled and the {kube,openshift,oauth}-encryption-config cmd line args have not been provided, generate new ones, encrypt target's resources and put them in etcd and in the filesystem
• ensure no control-plane rollouts occur after recert

[openshift-ci-robot]

@mresvanis: This pull request references MGMT-17267 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.18.0" version, but no target version was set.

In response to this:

This PR adds OpenShift etcd encryption support for recert:

• enable recert to read/edit/write OCP encrypted resources using the current encryption keys, during recertify and ocp_postprocess
  • AES-GCM
  • AES-CBC
• rotate the encryption keys
• encrypt all resources with the newly created encryption keys before committing to etcd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mresvanis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mresvanis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@mresvanis: This pull request references MGMT-17267 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.18.0" version, but no target version was set.

In response to this:

This PR adds OpenShift etcd encryption support for recert:

• enable recert to read/edit/write OCP encrypted resources using the current encryption keys, during recertify and ocp_postprocess
  • AES-GCM
  • AES-CBC
• rotate the encryption keys
• encrypt all resources with the newly created encryption keys before committing to etcd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@mresvanis: This pull request references MGMT-17267 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.18.0" version, but no target version was set.

In response to this:

This PR adds OpenShift etcd encryption support for recert:

• enable recert to read/edit/write OCP encrypted resources using the current encryption keys, during recertify and ocp_postprocess
  • AES-GCM
  • AES-CBC
• rotate the encryption keys
• encrypt resources to be encrypted with the newly created encryption keys before committing to etcd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@mresvanis: This pull request references MGMT-17267 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.18.0" version, but no target version was set.

In response to this:

This PR adds OpenShift etcd encryption support for recert:

• enable recert to read/edit/write OCP encrypted resources using the current encryption keys, during recertify and ocp_postprocess
  • AES-GCM
  • AES-CBC
• rotate the encryption keys
• encrypt resources to be encrypted with the newly created encryption keys before committing to etcd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[omertuc]

Very cool, mostly nits and one very annoying request - to switch to openssl and drop the dependencies, for FIPS reasons

[omertuc]

Is there any reason why we ask the user to provide us with the encryption config for kube-apiserver while we fetch the encryption config for openshift-apiserver and the oauth-apiserver on our own? What stops us from simply fetching the encryption config for kube-apiserver, instead of asking the user to provide us with this config?

[mresvanis]

What stops us from simply fetching the encryption config for kube-apiserver, instead of asking the user to provide us with this config?

The openshift-kube-apiserver/encryption-config is a Secret, which is encrypted. That leads us to the filesystem encryption-config file (under /etc/kubernetes/static-pod-resources/kube-apiserver-pod-<latest>/secrets/encryption-config/encryption-config) to fetch all the encryption details for the kube-apiserver encrypted resources (i.e. Secrets and ConfigMaps) from. Because it's a file and we're using recert mostly in a container, where we mount everything we need from the filesystem, I thought it best to have a path argument (which I will refactor to also accept the content of that file, as suggested here). Does this make sense?

[omertuc]

What stops us from simply fetching the encryption config for kube-apiserver, instead of asking the user to provide us with this config?

The openshift-kube-apiserver/encryption-config is a Secret, which is encrypted. That leads us to the filesystem encryption-config file (under /etc/kubernetes/static-pod-resources/kube-apiserver-pod-<latest>/secrets/encryption-config/encryption-config) to fetch all the encryption details for the kube-apiserver encrypted resources (i.e. Secrets and ConfigMaps) from. Because it's a file and we're using recert mostly in a container, where we mount everything we need from the filesystem, I thought it best to have a path argument (which I will refactor to also accept the content of that file, as suggested here). Does this make sense?

Yep, that makes perfect sense. Could you please explain that in a (non-documenting //) comment above the encryption_config parameter?

And add a reference to the /etc/kubernetes/static-pod-resources/kube-apiserver-pod-<latest>/secrets/encryption-config/encryption-config in the yes-documenting (///) part of the parameter comment

[openshift-ci-robot]

@mresvanis: This pull request references MGMT-17267 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.18.0" version, but no target version was set.

In response to this:

This PR adds OpenShift etcd encryption support for recert:

• enable recert to read/edit/write OCP encrypted resources using the current encryption keys, during recertify and ocp_postprocess
  • AES-GCM (openssl doesn't support AES-GCM, so not FIPS compliant anyway, we'll use Rust libs)
  • AES-CBC (via openssl for FIPS compliance)
• use seed encryption config to decrypt resources
• use target encryption config to encrypt resources before committing to etcd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@mresvanis: This pull request references MGMT-17267 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.18.0" version, but no target version was set.

In response to this:

This PR adds OpenShift etcd encryption support for recert:

• enable recert to read/edit/write OCP encrypted resources using the current encryption keys, during recertify and ocp_postprocess
  • AES-GCM (openssl doesn't support AES-GCM, so not FIPS compliant anyway, we'll use Rust libs)
  • AES-CBC (via openssl for FIPS compliance)
• use seed encryption config to decrypt resources during cert regeneration and OCP post-processing
• use target encryption config to encrypt resources before committing to etcd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@mresvanis: This pull request references MGMT-17267 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.18.0" version, but no target version was set.

In response to this:

This PR adds OpenShift etcd encryption support for recert:

• recert now auto-discovers the seed encryption config and reads OCP encrypted resources using those keys, during recertify and ocp_postprocess
  • supports AES-GCM (openssl doesn't support AES-GCM, so not FIPS compliant anyway, we'll use Rust libs)
  • supports AES-CBC (via openssl for FIPS compliance)
• add cmd line args for target's {kube,openshift,oauth}-apiserver encryption configs to keep target's keys and encrypt all resources with those before committing to etcd
• when seed encryption is enabled and the {kube,openshift,oauth}-encryption-config cmd line args have not been provided, generate new ones, encrypt target's resources and put them in etcd and in the filesystem
• ensure no control-plane rollouts occur after recert

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[mresvanis]

/test e2e-aws-ovn-single-node-recert-parallel

[omertuc]

Amazing work, mostly superficial comments

[mresvanis]

/test baremetalds-sno-recert-cluster-rename

[omertuc]

Thanks, small change

[omertuc]

/lgtm

[mresvanis]

/test e2e-aws-ovn-single-node-recert-parallel

[mresvanis]

@omertuc I'm not yet sure about what's happening with the conformance tests, but I believe it's related to OCP 4.18 and not this PR.

[omertuc]

Compare with #168

[mresvanis]

Compare with #168

We have almost the same results, PR 168:

# pull-ci-rh-ecosystem-edge-recert-main-e2e-aws-ovn-single-node-recert-parallel
: [sig-architecture] platform pods in ns/openshift-kube-apiserver should not exit an excessive amount of times expand_more	0s
: [sig-architecture] platform pods in ns/openshift-kube-controller-manager should not exit an excessive amount of times expand_more	0s
: [sig-architecture] platform pods in ns/openshift-kube-scheduler should not exit an excessive amount of times expand_more	0s
: [sig-node] static pods should start after being created expand_more	0s
: [sig-api-machinery][Feature:APIServer][Late] kubelet terminates kube-apiserver gracefully extended [Suite:openshift/conformance/parallel] expand_more

vs this PR 160

# pull-ci-rh-ecosystem-edge-recert-main-e2e-aws-ovn-single-node-recert-parallel
: [sig-architecture] platform pods in ns/openshift-kube-apiserver should not exit an excessive amount of times expand_more	0s
: [sig-architecture] platform pods in ns/openshift-kube-controller-manager should not exit an excessive amount of times expand_more	0s
: [sig-architecture] platform pods in ns/openshift-kube-scheduler should not exit an excessive amount of times expand_more	0s
: [sig-api-machinery][Feature:APIServer][Late] kubelet terminates kube-apiserver gracefully extended [Suite:openshift/conformance/parallel] expand_more

I'm now retesting the serial conformance tests.

[mresvanis]

/test e2e-aws-ovn-single-node-recert-serial

[openshift-ci]

@mresvanis: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-single-node-recert-parallel	a03c0db	link	true	/test e2e-aws-ovn-single-node-recert-parallel
ci/prow/e2e-aws-ovn-single-node-recert-serial	a03c0db	link	true	/test e2e-aws-ovn-single-node-recert-serial

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.