Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kafka AWS IAM support for authenication #3992

Merged
merged 7 commits into from
Feb 2, 2024
Merged

Add Kafka AWS IAM support for authenication #3992

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4764458
add kafka aws iam to docs
famarting Jan 31, 2024
2419ef9
Merge branch 'v1.13' into kafka-aws-iam
msfussell Feb 1, 2024
b2cbc2d
Update daprdocs/content/en/reference/components-reference/supported-p…
msfussell Feb 1, 2024
74fcb49
Update daprdocs/content/en/reference/components-reference/supported-p…
msfussell Feb 1, 2024
48e2f46
Update daprdocs/content/en/reference/components-reference/supported-p…
msfussell Feb 1, 2024
7a6f384
Update daprdocs/content/en/reference/components-reference/supported-p…
msfussell Feb 1, 2024
33c57ff
Merge branch 'v1.13' into kafka-aws-iam
msfussell Feb 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 44 additions & 2 deletions ...ontent/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ spec:
| consumerID | N | Consumer ID (consumer tag) organizes one or more consumers into a group. Consumers with the same consumer ID work as one virtual consumer; for example, a message is processed only once by one of the consumers in the group. If the `consumerID` is not provided, the Dapr runtime set it to the Dapr application ID (`appID`) value. If a value for `consumerGroup` is provided, any value for `consumerID` is ignored - a combination of the consumer group and a random unique identifier will be set for the `consumerID` instead. | `"channel1"`
| clientID | N | A user-provided string sent with every request to the Kafka brokers for logging, debugging, and auditing purposes. Defaults to `"namespace.appID"` for Kubernetes mode or `"appID"` for Self-Hosted mode. | `"my-namespace.my-dapr-app"`, `"my-dapr-app"`
| authRequired | N | *Deprecated* Enable [SASL](https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer) authentication with the Kafka brokers. | `"true"`, `"false"`
| authType | Y | Configure or disable authentication. Supported values: `none`, `password`, `mtls`, or `oidc` | `"password"`, `"none"`
| authType | Y | Configure or disable authentication. Supported values: `none`, `password`, `mtls`, `oidc` or `awsiam` | `"password"`, `"none"`
| saslUsername | N | The SASL username used for authentication. Only required if `authType` is set to `"password"`. | `"adminuser"`
| saslPassword | N | The SASL password used for authentication. Can be `secretKeyRef` to use a [secret reference]({{< ref component-secrets.md >}}). Only required if `authType is set to `"password"`. | `""`, `"KeFg23!"`
| saslMechanism | N | The SASL Authentication Mechanism you wish to use. Only required if `authType` is set to `"password"`. Defaults to `PLAINTEXT` | `"SHA-512", "SHA-256", "PLAINTEXT"`
Expand All @@ -92,6 +92,12 @@ spec:
| oidcClientSecret | N | The OAuth2 client secret that has been provisioned in the identity provider: Required when `authType` is set to `oidc` | `"KeFg23!"` |
| oidcScopes | N | Comma-delimited list of OAuth2/OIDC scopes to request with the access token. Recommended when `authType` is set to `oidc`. Defaults to `"openid"` | `"openid,kafka-prod"` |
| oidcExtensions | N | Input/Output | String containing a JSON-encoded dictionary of OAuth2/OIDC extensions to request with the access token | `{"cluster":"kafka","poolid":"kafkapool"}` |
| awsRegion | N | The AWS region where the Kafka cluster is deployed to. Required when `authType` is set to `awsiam` | `us-west-1` |
| awsAccessKey | N | AWS access key associated with an IAM account. | `"accessKey"`
| awsSecretKey | N | The secret key associated with the access key. | `"secretKey"`
| awsSessionToken | N | AWS session token to use. A session token is only required if you are using temporary security credentials. | `"sessionToken"`
| awsIamRoleArn | N | IAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials. | `"arn:aws:iam::123456789:role/mskRole"`
msfussell marked this conversation as resolved.
Show resolved Hide resolved
| awsStsSessionName | N | Represents the session name for assuming a role. | `"MSKSASLDefaultSession"`
| schemaRegistryURL | N | Required when using Schema Registry Avro serialization/deserialization. The Schema Registry URL. | `http://localhost:8081` |
| schemaRegistryAPIKey | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Key. | `XYAXXAZ` |
| schemaRegistryAPISecret | N | When using Schema Registry Avro serialization/deserialization. The Schema Registry credentials API Secret. | `ABCDEFGMEADFF` |
Expand All @@ -107,7 +113,7 @@ The metadata `version` must be set to `1.0.0` when using Azure EventHubs with Ka

Kafka supports a variety of authentication schemes and Dapr supports several: SASL password, mTLS, OIDC/OAuth2. With the added authentication methods, the `authRequired` field has
been deprecated from the v1.6 release and instead the `authType` field should be used. If `authRequired` is set to `true`, Dapr will attempt to configure `authType` correctly
based on the value of `saslPassword`. There are four valid values for `authType`: `none`, `password`, `certificate`, `mtls`, and `oidc`. Note this is authentication only; authorization is still configured within Kafka.
based on the value of `saslPassword`. This are the valid values for `authType`: `none`, `password`, `certificate`, `mtls`, `oidc` and `awsiam`. Note this is authentication only; authorization is still configured within Kafka except for `awsiam` which can also drive authorization decisions configured in AWS IAM.
msfussell marked this conversation as resolved.
Show resolved Hide resolved

#### None

Expand Down Expand Up @@ -276,6 +282,42 @@ spec:
value: 0.10.2.0
```

#### AWS IAM

Authenticating with AWS IAM is supported with AWS Managed Streaming for Apache Kafka (MSK). Setting `authType` to `awsiam` uses AWS SDK to generate auth tokens to authenticate.
msfussell marked this conversation as resolved.
Show resolved Hide resolved
Note the only required metadata field is `awsRegion`, if no `awsAccessKey` and `awsSecretKey` are provided you can use AWS IAM roles for service accounts to have passwordless authentication to your Kafka cluster.
msfussell marked this conversation as resolved.
Show resolved Hide resolved

```yaml
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: kafka-pubsub-awsiam
spec:
type: pubsub.kafka
version: v1
metadata:
- name: brokers # Required. Kafka broker connection setting
value: "dapr-kafka.myapp.svc.cluster.local:9092"
- name: consumerGroup # Optional. Used for input bindings.
value: "group1"
- name: clientID # Optional. Used as client tracing ID by Kafka brokers.
value: "my-dapr-app-id"
- name: authType # Required.
value: "awsiam"
- name: awsRegion # Required.
value: "us-west-1"
- name: awsAccessKey # Optional.
value: <AWS_ACCESS_KEY>
- name: awsSecretKey # Optional.
value: <AWS_SECRET_KEY>
- name: awsSessionToken # Optional.
value: <AWS_SESSION_KEY>
- name: awsIamRoleArn # Optional.
value: "arn:aws:iam::123456789:role/mskRole"
- name: awsStsSessionName # Optional.
value: "MSKSASLDefaultSession"
```

### Communication using TLS

By default TLS is enabled to secure the transport layer to Kafka. To disable TLS, set `disableTls` to `true`. When TLS is enabled, you can
Expand Down
Loading