At-rest encryption proposal to protect secrets in database

design/encryption-spi.md

@@ -0,0 +1,166 @@
+# Encrypt / decrypt support (part II)
+
+* **Status**: Draft #1
+* **Github**: TODO
+
+## Motivation
+
+This document continues from [secure-credentials-store.md](secure-credentials-store.md) and presents the design proposal for a new SPI: Encryption SPI.
+
+Current Vault SPI can be used to protect a subset of secrets by delegating at-rest storage to an external secret storage.
+The solution has some limitations:
+
+* Only read-only access to the vault is supported, which makes the solution suitable only for managing secrets provisioned by the administrator.
+If a secret is generated by Keycloak at runtime, like client secrets, it cannot be stored to an external vault using the current Vault SPI.
+* Using Vault SPI is not transparent, it adds complexity for the administrator since configuration of Keycloak is split into two separate steps:
+provisioning the secret into external storage and configuring a reference to Keycloak.
+
+The initial plan included part 2 to introduce encryption and decryption support, which is now proposed in this document.
+In addition to addressing the limitations of the current Vault SPI, this proposal aims to better comply with OWASP Top 10 Proactive Controls [C2: Use Cryptography](https://top10proactive.owasp.org/the-top-10/c2-crypto/) and related regulatory requirements.
+
+
+## Overview
+
+Encryption SPI implements capability for Keycloak to offload encryption / decryption of secrets to an external service.
+From Keycloak administrator's perspective, the solution is transparent.
+Administrator configures secrets in Keycloak as usual, and Keycloak encrypts them before storing in the database.
+
+The SPI abstracts the encryption key management and encryption algorithms away from Keycloak while allowing key rotation and aspects to be managed at the external service with mechanism provided by the service.
+
+
+## Scope
+
+The SPI can be used to encrypt / decrypt any secrets that needs to be stored in database:
+
+* Secrets generated by Keycloak at runtime (e.g. client secret)
+* Secrets received from external sources (stored tokens from identity brokering)
+* Secrets provisioned by the administrator (e.g. LDAP credentials)
+
+Keycloak treats the encrypted data as an opaque blob.
+The SPI provider is responsible for choosing the data format.
+
+Encryption key management and encryption algorithms are not part of the SPI.
+These are managed by the provider or the external service that the provider interfaces with.
+
+
+## Design
+
+### Configuration
+
+Configuration of the SPI is done by the administrator on global Keycloak level using command line flags
+
+```
+--spi-encryption-<provider-id>-<property>=<value>
+```
+
+The details of the configuration are specific to the implementation of the SPI.
+
+
+### Data migration and compatibility
+
+When the SPI is enabled for the first time, the database may contain plaintext secrets unless it is a new installation.
+It is also possible that new secret types will be introduced by upgrade of Keycloak.
+Therefore, it should not be assumed that all secrets in database are encrypted immediately upon enabling the SPI.
+
+The SPI implementation will only encrypt the secrets lazily.
+Plaintext secrets gets encrypted only when new secrets get created or old ones get updated.
+
+Secrets are wrapped in object that contains metadata about the implementation of the SPI that was used to encrypt the data.
+
+```
+${enc:provider-id:encrypted-data}
+```
+
+This encoding allows recognizing if encryption SPI was used to store the particular secret.
+If the secret is read from the database in plaintext then the plaintext secret is returned directly (this allows for backward compatibility).
+If the secret is read from the database in encrypted form then the secret is decrypted with the provider before returning it.
+If the provider is not available then error is logged and exception is thrown.
+
+The `encrypted-data` is stored in base64 encoding.
+The data itself is opaque for Keycloak, including the encryption algorithm used for encryption and any possible metadata included in `encrypted-data`.
+These are managed internally by the SPI provider.
+
+> **TODO**: Q: Is lazy encryption acceptable as an approach?
+>
+> Lazy encryption limits the impact to code that handles secrets.
+> The problem is limited to providing and using encrypt/decrypt operations.
+>
+> It may be possible that this approach is too trivial and more complex solution is needed.
+> As an example, consider following use cases:
+>
+> (1) Administrator might want to encrypt all data at once when taking the SPI into use in existing installation.
+>
+> (2) Administrator might want to re-encrypt all data with new encryption key after key rotation.
+>
+> (3) Administrator might want to query for the status of the encryption: are all data encrypted, which encryption provider was used, which encryption keys are in use.
+>
+> (4) Administrator might want to decrypt all data.
+> This could be used to migrate data back to plaintext or to move to another encryption provider.
+> It could be used to recover from unseen issues.
+>
+> Supporting the above use cases would require much wider changes than presented in this document.
+> For example, there could be a "framework" to support marking which data fields are sensitive, making it possible to query and iterate over all sensitive fields in the database.
+
+### Interface
+
+```java
+public interface EncryptionServiceProvider {
+    Optional<byte[]> encrypt(byte[] plaintext);
+    Optional<byte[]> decrypt(byte[] ciphertext);
+}
+```
+
+### Supported fields
+
+Following fields are proposed to be encrypted by the SPI.
+Fields can be included incrementally in the future, and not all fields need to be covered at once.
+
+* Client secret generated by Keycloak.
+* Client secret configured by administrator for identity brokering.
+* External tokens stored in the database by Keycloak when "Store tokens" is configured for identity brokering.
+* Realm keys generated by Keycloak.
+* OTP shared secrets generated by Keycloak.
+* LDAP credentials configured by administrator.
+* SMTP credentials configured by administrator.
+
+There is some overlap with the Vault SPI.
+It is not anticipated that both the Vault SPI and Encryption SPI will be used simultaneously.
+The justification for the overlap are discussed in the Motivation section.
+
+Encryption SPI cannot be used for fields that need to remain searchable in the database.
+This can include fields like username, email, or other fields that can be considered as privacy-sensitive but need to be searchable.
+
+
+### Performance
+
+> **TODO**: Q: What is the expected performance impact?
+>
+> Based on the current proposal, the encryption service is invoked every time a secret is stored or retrieved from the database.
+>
+> There is no performance issue while the secret is in use, since it's kept in plaintext in memory.
+> However, if secrets are frequently retrieved or stored and the encryption service is remote, the performance impact could be significant.
+>
+> Considering the current usage patterns and in-memory caching, will the performance be acceptable?
+> For example, could this approach cause a spike in requests to the encryption service when the server starts up?
+> Should we consider a more advanced solution where most encryption is handled locally, reducing the number of calls to the encryption service?
+
+### Default implementation
+
+The encryption SPI is disabled by default.
+When disabled there is no impact on the existing functionality of Keycloak or the format of the data stored in the database.
+
+> **TODO**: Q: Should Keycloak include default provider for Encryption SPI?
+>
+> Some possible alternatives:
+>
+> (1) No default provider implementation: it is assumed providers will be implemented by the community.
+>
+> (2) Implement "plaintext" provider for testing that does not encrypt the data but returns it as is.
+>
+> (3) Implement provider for HashiCorp Vault that encrypts the data using Vault Transit secrets engine.
+>
+> (4) Implement a local provider that encrypts secrets using Java Cryptography support.
+>
+> Local provider would require relatively complicated new functionality Keycloak, including managing encryption keys and algorithms.
+> HashiCorp Vault Transit secrets engine could be good candidate for implementation, as it is able to manage encryption keys and supports various encryption algorithms.
+> The REST API is relatively simple to use directly, without adding full blown Vault SDK client library.