Add first class support for client-side encryption of repositories

[andrross]

The repository interface in OpenSearch enables storing snapshot data in an external object store. It is implemented as a plugin interface, and the OpenSearch bundle by default ships with multiple plugin implementations (e.g. Google Cloud Storage, Amazon S3, Microsoft Azure). These object stores may offer server-side encryption, but OpenSearch itself offers no out-of-the-box mechanism to enable client-side encryption of the snapshot data. Both Amazon OpenSearch Service and Aiven have implemented client-side encryption, so there is clearly a need for this feature. The request here is to offer first class support for this feature within the OpenSearch Project.

We have multiple projects in flight (remote-backed index, searchable snapshots, etc) that are building on the repository interface. All of these stand to benefit from a client-side encryption feature.

Describe the solution you'd like
The Aiven implementation is an open source plugin that essentially wraps the repository interface, allowing users to compose the encryption plugin with the object store implementation of their choice. An obvious option here is to bring this plugin along side the other object store plugins that are bundled in the OpenSearch distribution.

Describe alternatives you've considered
There might be benefits to building client-side encryption more natively into the repository interface. The compose-ability of the Aiven implementation offers great flexibility, but it does mean that users must be aware of the plugin and configure it accordingly. Building it more directly into the interface could have usability benefits?

[willyborankin]

Hi @nknize, @andrross, and @reta I'm the author of the encrypted-repository-opensearch plugin.
Below you will find some thoughts on what I believe OpenSearch is missing as a default solution for plugins that use encryption.
Beside I would like to invite opensearch-project/security team: @scrawfor99 , @peternied, @cwperks, @cliu123, @DarshitChanpura, @davidlago to discuss this solution as well.

All repository plugins, as you all know, just work with stream chunks, which the OpenSearch kernel sends to 3rd-party systems like HDFS, file systems, clouds, etc. To avoid different configurations of how they should encrypt data. I suggest using the KEK/DEK approach.

What is DEK/KEK

Data Encryption Key (DEK)
The key used to encrypt data itself is called a data encryption key (DEK).

Key encryption keys (KEK)
The DEK is encrypted (also known as wrapped) by a key encryption key (KEK). The process of encrypting a key with another key is known as envelope encryption.

KEK management rules

• Store KEKs centrally
• You could use a single KEK to wrap all DEKs that are responsible for that workload's encryption.
• Rotate keys regularly, and also after a suspected incident

DEK management rules

• Generate DEKs locally
• When stored, always ensure DEKs are encrypted at rest
• For easy access, store the DEK near the data that it encrypts
• Generate a new DEK every time you write the data. This means you don't need to rotate the DEKs
• Do not use the same DEK to encrypt data from two different users
• Use a strong algorithm such as 256-bit Advanced Encryption Standard (AES) in Galois Counter Mode (GCM).

For that, OpenSearch needs to have a Key Manager Service, which plugins will use to generate DEK keys.

Key Management Service

The key management service is a central service which OpenSearch provides to plugins to use for encryption. It can be integrated with 3rd party KMS e.g. Amazon, Google etc. There is only one responsibility for the key management service: to generate, encrypt, and decrypt DEK keys or additional data on demand. It can be configured to use different key management systems using names, so that plug-ins can use different key management systems.
The configuration of the key management service needs to be done via the OpenSearch key store and shouldn't affect the bootstrapping of nodes or cluster formation.
The default implementation uses RSA-256 bit keys to encrypt/decrypt AES-256 bit DEK keys and act as KEK.

The POC interface:

interface KeyManagerServiceProvider {

   default KeyManagerService keyManagerService() {
       return  keyManagerService(“default”);
   }

   KeyManagerService keyManagerService(final String name);

}

interface KeyManagerService {
   
   Supplier<SecretKey> dekGenerator();

   byte[] decrypt(final byte[] data)

   byte[] encrypt(final byte[] data);

}

Keys rotation:

• The KEK/DEK model forces to use multiple DEK keys for encryption and decryption so rotation for DEK keys is working out of the box
• KEK keys rotation depends on the key management service configuration. In case of 3rd party KMS providers it is up to the provider. For the default key management service the rules are the same as for all other security settings: update settings on all nodes in the cluster and call /_nodes/reload_secure_settings or /_nodes/<node_id>/reload_secure_settings on the management node.

Default Key Management Service Configuration Key Store Settings

• key.management.service.default.public.key - RSA 256-bit public key
• key.management.service.default.private.key - RSA 256-bit private key
• key.management.service.default.dek.encryption.algorithm - RSA encryption/decryption algorithm

Key Management Service Configuration Key Store Settings For 3rd Party Providers

• key.management.service.<name>.<provider>.* - set up all necessary 3rd provider settings to use it

All management services can be configured independently.

How it will look like for plugins

In case of Key Management Service all plugins will use the same rules/recommendations for encryption:
the main is "Generate a new DEK every time you write the data" which means DEK key per stream.

E.g. in case of encrypted-repository-opensearch all you need to add is only KMS name which will be use by the plugin to generate DEKs:

      PUT _snapshot/repository_name
        {
          "type": "encrypted",
          "settings": {
            "storage_type": "fs",   
            "key_manager_name: "some_kms",
            "location": "/mount/backups/my_fs_backup_location"            
          }
        }

and during configuration of the repository ClusterService will return information about the OpenSearch key manager service the plugin needs to use and after that generate DEK for each stream.

If I'm missing something fill free to point out.

P.S. In the end I would like to address some comments I read in PRs about the encrypted-repository-opensearch plugin which I found unfair and will address here:

 It does not support partial encryption

TBH, I do not understand what partial encryption is. If it is about blocking encryption, I am afraid it was compromised; e.g., AES/CBC are not recommended for such encryption. Yes, it is possible to implement a custom solution for such case which again needs to be proven stable and will look like CBC. If it is about AES/GCS and the limitation of the stream size up to 64GB due to the IV and the fact that it must be renewed, well, it's a natural limitation of AES/GCM which is solved via the hard limit of a chuck size of 64 GB in the plugin since the default chuck size for all clouds is 5 TB, and in this case the whole stream will be compromised.

It does not support chucking 

It does support chunking this solution was select this way since it is responsibility of OpenSearch (in the past ES) to split streams into chunks which it does perfectly fine.

It is a custom wrapper around java encryption library

I do not understand what a custom wrapper in this case. The plugin uses classical algorithms and cryptographic primitives e.g. AES 256-bit keys in Galois Counter Mode which were proved to be stable for such tasks. The only reason why BC was selected is that the problem of JDK which does not support streaming with more than 2GB and it is only one well known and trusted solution in opensource world. It is possible to use AWS or Google libraries, but I do not know how popular are they and it is up to the OpenSearch team.If it is about storing IV and encrypted data at the beginning of the stream, again, it is just an optimization. In this case, you don't need to do an additional network round trip.

 it is tightly coupled with repository plugin

Yes and it is only right solution since OpenSearch core controls when it wants to send data somewhere. Moving such CPU bounded tasks to the core will affect performance of OpenSerach.

[reta]

@willyborankin thanks a lot for elaborate details

In case of Key Management Service all plugins will use the same rules/recommendations for encryption:
the main is "Generate a new DEK every time you write the data" which means DEK key per stream.

It totally make sense from the perfect encryption standpoint, what information (if any) has to be stored with the snapshot (key provider? key name/id? nothing?) in order to decrypt it?

[willyborankin]

@willyborankin thanks a lot for elaborate details

In case of Key Management Service all plugins will use the same rules/recommendations for encryption:
the main is "Generate a new DEK every time you write the data" which means DEK key per stream.

It totally make sense from the perfect encryption standpoint, what information (if any) has to be stored with the snapshot (key provider? key name/id? nothing?) in order to decrypt it?

Hi @reta, I added KEK/DEK management rules to the description.

[andrross]

Thanks @willyborankin!

it is tightly coupled with repository plugin

@willyborankin I think the key point here is that KeyManagerServiceProvider and KeyManagerService in your proposal need not be tightly coupled to the repository plugin and could be reused to encrypt other data at rest within OpenSearch in the future. What is your thinking with regard to this? Where would this new service provider interface be defined?

It does not support partial encryption

TBH, I do not understand what partial encryption is

I'm also not clear here. Is this referring to the unimplemented readBlob override that fetches a random offset of a file?

Tagging @vikasvb90 to share his thoughts here.

[willyborankin]

Hi @andrross,

Thanks @willyborankin!

it is tightly coupled with repository plugin

@willyborankin I think the key point here is that KeyManagerServiceProvider and KeyManagerService in your proposal need not be tightly coupled to the repository plugin and could be reused to encrypt other data at rest within OpenSearch in the future. What is your thinking with regard to this? Where would this new service provider interface be defined?

I think key manager service should an internal OpenSearch service, sorry my explanation is not clear.
In this case, KEK keys are in the one place and plugin use only DEKs.
And the injection of the KMSProvider should be done via interface which plugin needs to extend like for all other cases when you develop a new plugin, e.g., DiscoveryPlugin, ClusterPlugin, etc.

It does not support partial encryption

TBH, I do not understand what partial encryption is

I'm also not clear here. Is this referring to the unimplemented readBlob override that fetches a random offset of a file?

Ahh AFAIK AES/CTR support random access and you can encrypt and decrypt it in parallel, GCM is about integrity ans speed. This check was added due to the fact that in ES it used (I do not know how it looks now) for searchable snapshots.

Tagging @vikasvb90 to share his thoughts here.

[peternied]

Thanks for writing this up @willyborankin there is a lot to like with this approach, couple areas for your consideration:

Protect decryption?

I believe in OpenSearch plugin ecosystem runtime if you can find the data you can ask for the key and decrypt it with this interface. Could we protect the key generation process so the keys only match the same plugin, would that be useful?

Static declarations vs dynamically providers?

As outlined, OpenSearch configuration settings know how to find the KMS providers, what would you think about having KMS providers be dynamically on/off 'lined? Having (de)encryption fall out of sync between nodes could create some painful operator scenarios, could this be mitigated?

[vikasvb90]

@willyborankin
Let me start by addressing your comments

TBH, I do not understand what partial encryption is.

This is not about using block based algorithms or specifically limiting to any deprecated algorithm which supports this. All standard committing algorithms used in AWS encryption SDK like AES/GCM can be used for this. As @andrross pointed out, this is about randomly encrypting or decrypting a part of the content.

GCM is about integrity ans speed. This check was added due to the fact that in ES it used (I do not know how it looks now) for searchable snapshots.

You can look at this PR. It uses frame based encryption. Frame is the smallest unit of encryption or decryption of default size 4KB. Frame based encryption is the default mode for encryption and committing algos like AES/GCM. Both ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY and ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY_ECDSA_P384 work. Other non committing algorithms like IV algos are not recommended for use which is what I think you are also trying to point.

We cannot disable random encryption or decryption as it is required for features like searchable snapshots. We do need a solution for this. This can be achieved in frame encryption by ensuring that encrypt and decrypt operations always happen along frame boundaries, metadata headers which consist of crypto materials are added only while encrypting the first frame of the content and frame numbers are appropriately assigned. This is what the PR above ensures. As long as these conditions are met, you don't have to encrypt or decrypt the entire content just to access a small part of it.

Now, coming to your repository based design, I have a number of concerns with this approach.

1. Partial encryption (or random encryption or decryption) cannot be supported with existing abstractions of repository plugin.
2. You have built a whole new delegation layer just to fit encryption layer with storage layer in order to specifically support full content encryption and decryption. Other repository behaviors like deleteBlobs, children, listBlobs, etc. do not really apply in the context of encryption but in the context of storage.
3. Non-repository use cases can't leverage encryption/decryption capabilities. By decoupling repository with encryption, we can easily allow some of the potential features to be built. Basically you are free to add any customization. For e.g.,
   i) Encryption of local indexes.
   ii) Selective encryption. Since encryption is known to consume a lot of compute power, users might demand this where they are able to mark certain fields as sensitive and allow only those fields to be encrypted.
4. With this tight coupling, you always need to encrypt everything. This means that you cannot selectively avoid encrypting metadata files like checkpoint and snapshot metadata files. This is a performance overhead.

Yes and it is only right solution since OpenSearch core controls when it wants to send data somewhere. Moving such CPU bounded tasks to the core will affect performance of OpenSerach.

Not sure how tight coupling is ensuring less CPU. It is solving a specific design issue of encrypting whole content and being agnostic of it during transfers. In fact, it has an additional performance overhead where there is a forced need of encrypting everything which is not there in decoupled flavor where you can selectively encrypt or decrypt. May be you are referring to encryption being applied at file system where local IO is more frequent like in case of segment merging, tlog/ckp files, etc. as compared to snapshot transfers. If yes, then I am not again suggesting it to be built here. In my opinion it should not be tied to any storage system - local or remote and should be used as a tool to just transform data as per the need. It should be agnostic of location of source or target content or how we plan to use the resultant content.

Also, I don't think it is a good idea to build KMS as a service within OpenSearch. It should be an extensible plugin where community is free to implement their own versions of KMS. I also don't see the need of adding key stores within cluster state. This means that for every key rotation, this would require a cluster state update. In my opinion, cluster state can just keep immutable key provider settings.

Generate DEKs locally
It can be integrated with 3rd party KMS e.g. Amazon, Google etc. There is only one responsibility for the key management service: to generate, encrypt, and decrypt DEK keys

About generating DEKs locally, both are contradicting statements unless I am missing something. And you mentioned Generate a new DEK every time you write the data. This means you don't need to rotate the DEKs. Since generate data key pair is a call to third party system, I don't think this is ideal since there are hard limits imposed by systems on the usage and there is additional network round trip on every write.

[willyborankin]

Thank you for the answer.

@willyborankin Let me start by addressing your comments

TBH, I do not understand what partial encryption is.

This is not about using block based algorithms or specifically limiting to any deprecated algorithm which supports this. All standard committing algorithms used in AWS encryption SDK like AES/GCM can be used for this. As @andrross pointed out, this is about randomly encrypting or decrypting a part of the content.

GCM is about integrity ans speed. This check was added due to the fact that in ES it used (I do not know how it looks now) for searchable snapshots.

You can look at this PR. It uses frame based encryption. Frame is the smallest unit of encryption or decryption of default size 4KB. Frame based encryption is the default mode for encryption and committing algos like AES/GCM. Both ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY and ALG_AES_256_GCM_HKDF_SHA512_COMMIT_KEY_ECDSA_P384 work. Other non committing algorithms like IV algos are not recommended for use which is what I think you are also trying to point.

We cannot disable random encryption or decryption as it is required for features like searchable snapshots. We do need a solution for this. This can be achieved in frame encryption by ensuring that encrypt and decrypt operations always happen along frame boundaries, metadata headers which consist of crypto materials are added only while encrypting the first frame of the content and frame numbers are appropriately assigned. This is what the PR above ensures. As long as these conditions are met, you don't have to encrypt or decrypt the entire content just to access a small part of it.

By random encryption I think you mean random access file right?
I would verify existing stable algos for encryption in this case which supports random access.

Now, coming to your repository based design, I have a number of concerns with this approach.

1. Partial encryption (or random encryption or decryption) cannot be supported with existing abstractions of repository plugin.

2. You have built a whole new delegation layer just to fit encryption layer with storage layer in order to specifically support full content encryption and decryption. Other repository behaviors like deleteBlobs, children, listBlobs, etc. do not really apply in the context of encryption but in the context of storage.

It was done this way just to avoid changing existing repositories: S3, HDFS, GCS, Azure, and FS. In your solution, you need to change all exiting repository plugins, and each needs to be configured to use a different key store management with its own key store. Such solution is very difficult to support from the OPS point of view.

3. Non-repository use cases can't leverage encryption/decryption capabilities. By decoupling repository with encryption, we can easily allow some of the potential features to be built. Basically you are free to add any customization.  For e.g.,
   i) Encryption of local indexes.
   ii) Selective encryption. Since encryption is known to consume a lot of compute power, users might demand this where they are able to mark certain fields as sensitive and allow only those fields to be encrypted.

4. With this tight coupling, you always need to encrypt everything. This means that you cannot selectively avoid encrypting metadata files like checkpoint and snapshot metadata files. This is a performance overhead.

I got your point about special fields thank you. IMHO this is about CSE :-).
AFAIK most of the case just encrypt-me-everything, I'm not sure that Selective encryption is so common.

Yes and it is only right solution since OpenSearch core controls when it wants to send data somewhere. Moving such CPU bounded tasks to the core will affect performance of OpenSerach.

Not sure how tight coupling is ensuring less CPU. It is solving a specific design issue of encrypting whole content and being agnostic of it during transfers. In fact, it has an additional performance overhead where there is a forced need of encrypting everything which is not there in decoupled flavor where you can selectively encrypt or decrypt. May be you are referring to encryption being applied at file system where local IO is more frequent like in case of segment merging, tlog/ckp files, etc. as compared to snapshot transfers. If yes, then I am not again suggesting it to be built here. In my opinion it should not be tied to any storage system - local or remote and should be used as a tool to just transform data as per the need. It should be agnostic of location of source or target content or how we plan to use the resultant content.

Also, I don't think it is a good idea to build KMS as a service within OpenSearch. It should be an extensible plugin where community is free to implement their own versions of KMS. I also don't see the need of adding key stores within cluster state. This means that for every key rotation, this would require a cluster state update. In my opinion, cluster state can just keep immutable key provider settings.

I checked one more time my description and did not find anything related to the storing KMS in the cluster state sorry.

Generate DEKs locally
It can be integrated with 3rd party KMS e.g. Amazon, Google etc. There is only one responsibility for the key management service: to generate, encrypt, and decrypt DEK keys

About generating DEKs locally, both are contradicting statements unless I am missing something. And you mentioned Generate a new DEK every time you write the data. This means you don't need to rotate the DEKs. Since generate data key pair is a call to third party system, I don't think this is ideal since there are hard limits imposed by systems on the usage and there is additional network round trip on every write.

Do you mean KEK, which a 3rd-party system stores? KEK/DEK assumes that you use KEK for encryption of DEK keys. And getting the latest info about KEK is a remote call since it is the responsibility of 3rd-party systems, while DEK generates via any key generator you have in any library. Besides, storing DEKs together with KEKs is not a good idea.

[willyborankin]

Thank for you,

Thanks for writing this up @willyborankin there is a lot to like with this approach, couple areas for your consideration:

Protect decryption?

I believe in OpenSearch plugin ecosystem runtime if you can find the data you can ask for the key and decrypt it with this interface. Could we protect the key generation process so the keys only match the same plugin, would that be useful?

Technically, it is possible e.g., by creating a new KMS and changing its state via listener if something has changed in the key store. AFAIK, OpenSearch core supports listeners but I'm not sure about it.

Static declarations vs dynamically providers?

As outlined, OpenSearch configuration settings know how to find the KMS providers, what would you think about having KMS providers be dynamically on/off 'lined? Having (de)encryption fall out of sync between nodes could create some painful operator scenarios, could this be mitigated?

Good point! Sometimes it is necessary block something and after that rotate keys.

[vikasvb90]

Thanks for the reply @willyborankin

I would verify existing stable algos for encryption in this case which supports random access.

I have verified random access with AES/GCM committing algos. Carried out few perfs as well. I have added few tests to ensure the integrity. Are you suggesting verifying this against any other stable algo?

It was done this way just to avoid changing existing repositories: S3, HDFS, GCS, Azure, and FS. In your solution, you need to change all exiting repository plugins, and each needs to be configured to use a different key store management with its own key store. Such solution is very difficult to support from the OPS point of view.

That is exactly why I am not inclined to tie it to repository abstraction and keep it as a separate transformation layer. Since there is no straightforward way to accommodate it at repository level as you also mentioned, a feature will selectively encrypt or decrypt content before invoking repository transfers.

AFAIK most of the case just encrypt-me-everything, I'm not sure that Selective encryption is so common.

There are existing products like this which already offer these capabilities. Even if you feel it is not a good reason, then also we have existing random encryption/decryption use cases like searchable snapshots and multi-part parallel transfers because of which we need to decouple repository with crypto. This is because of the point which you also mentioned where accommodating random access would require changes across all repository plugins and may not be a practical approach.

KEK keys rotation depends on the key management service configuration. In case of 3rd party KMS providers it is up to the provider. For the default key management service the rules are the same as for all other security settings: update settings on all nodes in the cluster and call /_nodes/reload_secure_settings or /_nodes/<node_id>/reload_secure_settings on the management node.

This line mentions key rotation and on its event updating cluster settings. This is why I mentioned that we shouldn't be updating cluster state or settings on rotation as this is an expensive operation. And since we are caching encrypted or generated key pairs, it is ok to have individual nodes maintain their own data keys.

Do you mean KEK, which a 3rd-party system stores? KEK/DEK assumes that you use KEK for encryption of DEK keys. And getting the latest info about KEK is a remote call since it is the responsibility of 3rd-party systems, while DEK generates via any key generator you have in any library. Besides, storing DEKs together with KEKs is not a good idea.

Got it! Just to further confirm that when you say DEK are you referring to raw version of encrypted keys (KEK in your case)? I agree that we shouldn't be putting raw keys along with encrypted content. I am not suggesting this either. This beats the whole point of encryption. Now, I think you are suggesting to use encrypt abstraction of key providers which basically encrypt the data keys or used in wrapping keys. Why not use generateDataPair itself which provides both data key and encrypted key as a key pair and saves the overhead of generating keys locally? Locally generating keys is anyways not sufficient since we anyways need to add encrypted version of the key along with the content. encrypt is generally used for wrapping use cases where multiple keys are used for encryption/decryption.

[vikasvb90]

@willyborankin This is a meta issue which has all the PRs of crypto. Please take a look and let me know if anything is missing #7229 .

[andrross]

then also we have existing random encryption/decryption use cases like searchable snapshots and multi-part parallel transfers because of which we need to decouple repository with crypto

@vikasvb90 Can you give more detail here? Why do we have to reimplement the crypto integration logic in every feature that uses it in order to support these capabilities?

Also, I'll say again I'm strongly in favor of encrypting everything written to the repository, regardless of how the code is architected. If we exempt certain metadata files, for example, then I think every operator of OpenSearch that requires client-side encryption will have to do a thorough security audit to ensure the plain text metadata doesn't expose any information that they deem to be sensitive. Every update of OpenSearch may require such an audit as well as the contents of the metadata may change over time. It seems much simpler to side step these questions by encrypting everything.

[willyborankin]

@willyborankin This is a meta issue which has all the PRs of crypto. Please take a look and let me know if anything is missing #7229 .

Thanks @vikasvb90. I will leave my comments during the week in your PR.

[andrross]

Just want to summarize some offline conversations here. I think we agree on the following architectural points:

• the key provider should be pluggable and not directly tied to a repository
• all data written to the repository will be encrypted (we can add the ability to opt-out specific files from encryption later if needed for performance or other reasons, but we’ll start with the simple approach)
• the heavy-lifting of doing encryption and decryption should be implemented in a common layer

We'll continue to iterate on the PRs that @vikasvb90 has open and work toward consensus here.