docs: add essential Kafka client security parameters to bindings #56

dalelane · 2021-02-24T14:49:04Z

There are a huge number of Kafka client configs we could add to
server bindings, but I'm skeptical of the value of just copying
them all across. Most client config values have sensible
defaults that are suitable in most cases.
cf. https://kafka.apache.org/documentation/#consumerconfigs

Two values are essential for code generation with secured Kafka
clusters, however - identifying the security mechanism and
protocol to use. This commit adds these two most commonly-used
parameters to the Kafka server bindings. I suggest we start with
these and gauge usage before starting to add many more, as these
are enough to enable code generation for secured Kafka clusters.

Closes: asyncapi/spec#466

Signed-off-by: Dale Lane [email protected]

There are a huge number of Kafka client configs we could add to server bindings, but I'm skeptical of the value of just copying them all across. Most client config values have sensible defaults that are suitable in most cases. cf. https://kafka.apache.org/documentation/#consumerconfigs Two values are essential for code generation with secured Kafka clusters, however - identifying the security mechanism and protocol to use. This commit adds these two most commonly-used parameters to the Kafka server bindings. I suggest we start with these and gauge usage before starting to add many more, as these are enough to enable code generation for secured Kafka clusters. Closes: asyncapi/spec#466 Signed-off-by: Dale Lane <[email protected]>

smoya · 2021-02-24T15:53:02Z

Hi @dalelane! Thanks for making this PR.
I'm wondering if we could add all those auth mechanisms as Security Schemas instead, so they can be reused. I see all those mechanisms are not only Kafka-oriented.

[Edit] I see @fmvilas suggested this already as per your notes in asyncapi/spec#466

dalelane · 2021-02-24T15:55:50Z

I'm wondering if we could add all those auth mechanisms as Security Schemas instead, so they can be reused. I see all those mechanisms are not only Kafka-oriented.

Could you give me a bit more detail about how that'd work, please? Or maybe an example of what this would look like in security schemes?

saslMechanism: 'SCRAM-SHA-512'
securityProtocol: 'SASL_PLAINTEXT'

smoya · 2021-02-24T16:13:44Z

Could you give me a bit more detail about how that'd work, please? Or maybe an example of what this would look like in security schemes?

We already have implemented one of the available SASL mechanisms: x509.
I think you could define a new schema for each mechanism, e.g. SASL SCRAM-SHA-512:

type: saslScramSha512

Or create a global one for all SHA, which means we would have x509 duplicated, not so ideal I think.

type: sasl
saslMechanism: ScramSha512

fmvilas · 2021-02-24T16:53:35Z

kafka/README.md


+Field Name | Type | Description | Applicability [default] | Constraints


If the whole "Constraints" column is empty, shouldn't we just get rid of it entirely?

I was doing that to avoid the merge conflict if both this and #55 get approved

Oh, I see. Thanks for the explanation 👍

fmvilas · 2021-02-24T16:54:19Z

kafka/README.md


+Field Name | Type | Description | Applicability [default] | Constraints
+---|:---:|:---:|:---:|---
+`saslMechamism` | string | Specifies the [`sasl.mechanism`](https://kafka.apache.org/documentation/#consumerconfigs_sasl.mechanism) config property that must be specified by Kafka clients connecting to this server. <br>Example values include `GSSAPI`, `PLAIN`, `OAUTHBEARER`, `SCRAM-SHA-256`, `SCRAM-SHA-512`  | OPTIONAL [`GSSAPI`] | -


Really the default value is GSSAPI? Isn't it PLAIN?

Also, shouldn't we have something like a NONE value to disable the saslMechanism?

I did that to mirror what Kafka does by default - which is GSSAPI

https://github.com/apache/kafka/blob/fc68c0fc9b1199b4eb63d9d486b012667c1d448f/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L32

But I think adding a NONE option for when using kafka instead of kafka-secure might make sense, and make a better default.

Yeah. In any case, if we move this to the Security Schemes Object, we would not need the NONE value anymore. Sorry, for the messed-up review. I realized about the other comments after I made the review.

no problem at all - it's been a useful thread, thanks to both you and @smoya !

fmvilas · 2021-02-24T16:59:21Z

I just saw Sergio's comment. Yeah, it's true it would make more sense to add this to Security Schemes. The way to proceed is similar. If something is missing on the Security Schemes Object, then feel free to add new keys, like saslMechanism and securityProtocol.

nictownsend · 2021-02-24T17:26:16Z

The problem with x509 is that it becomes overloaded - or at least for Kafka - in that you can have SSL based authentication, and SASL based. But there's no sasl x509 - so using sasl as a prefix on any new type would be misleading (unless x509 is documented as shared between concepts).

smoya · 2021-02-25T10:17:38Z

The problem with x509 is that it becomes overloaded - or at least for Kafka - in that you can have SSL based authentication, and SASL based. But there's no sasl x509 - so using sasl as a prefix on any new type would be misleading (unless x509 is documented as shared between concepts).

You are totally right @nictownsend. x509 would need to be a shared mechanism, in this case between TLS/SSL and SASL.
What if we just create one schema for each mechanism?

GSSAPI
OAUTHBEARER (This is already available under the oauth2 schema iirc)
SCRAM-SHA-256 (Not sure if it's a good idea to reuse userPassword schema, same for the next one)
SCRAM-SHA-512

dalelane · 2021-02-25T11:42:47Z

That's been really useful, thanks @smoya @nictownsend

I think where we end up is:

AsyncAPI: server `protocol`	AsyncAPI: security scheme `type`	meaning encryption?	meaning auth?	maps to Kafka client parameter `sasl.mechanism`	maps to Kafka client parameter `security.protocol`
`kafka`		no	no	unset	`PLAINTEXT`
`kafka`	`plain` ^*	no	yes, using SASL/PLAIN	`PLAIN`	`SASL_PLAINTEXT`
`kafka`	`scramSha256` ^*	no	yes, using SASL/SCRAM	`SCRAM-SHA-256`	`SASL_PLAINTEXT`
`kafka`	`scramSha512` ^*	no	yes, using SASL/SCRAM	`SCRAM-SHA-512`	`SASL_PLAINTEXT`
`kafka`	`oauth2`	no	yes, using OAuth	`OAUTHBEARER`	`SASL_PLAINTEXT`
`kafka`	`gssapi` ^*	no	yes, using GSSAPI	`GSSAPI`	`SASL_PLAINTEXT`
`kafka-secure`		yes	no	unset	`SSL`
`kafka-secure`	`plain` ^*	yes	yes, using SASL/PLAIN	`PLAIN`	`SASL_SSL`
`kafka-secure`	`scramSha256` ^*	yes	yes, using SASL/SCRAM	`SCRAM-SHA-256`	`SASL_SSL`
`kafka-secure`	`scramSha512` ^*	yes	yes, using SASL/SCRAM	`SCRAM-SHA-512`	`SASL_SSL`
`kafka-secure`	`oauth2`	yes	yes, using OAuth	`OAUTHBEARER`	`SASL_SSL`
`kafka-secure`	`gssapi` ^*	yes	yes, using GSSAPI	`GSSAPI`	`SASL_SSL`
`kafka-secure`	`X509`	yes	yes, using mutual TLS	unset	`SSL`

^* - a new security scheme type to be added

dalelane · 2021-02-25T15:50:21Z

Closing as I think we've agreed this is better captured in security schemes, as described in #56 (comment) so I've opened asyncapi/spec#502 for this

* feat: add SASL security scheme types for use by Kafka specs As discussed in asyncapi/bindings#56 this adds additional security scheme types. The motivation for adding them is to enable description of secured Kafka clusters, however the security protocols and mechanisms being added are not unique to Kafka, so this commit adds them as generic security schemes so they could be used by other protocols as well. Contributes to: #466 Signed-off-by: Dale Lane <[email protected]> * chore: update for grammar Moved the "and" to the end of the list. Signed-off-by: Dale Lane <[email protected]> * docs: additional SASL documentation Contributes to: #466 Signed-off-by: Dale Lane <[email protected]>

fmvilas reviewed Feb 24, 2021

View reviewed changes

dalelane mentioned this pull request Feb 25, 2021

feat: add SASL security scheme types for use by Kafka specs asyncapi/spec#502

Merged

dalelane closed this Feb 25, 2021

dalelane mentioned this pull request Feb 26, 2021

AsyncAPI open meeting, Tuesday Mar 2 2021 4PM UTC asyncapi/spec#504

Closed

magicmatatjahu mentioned this pull request Mar 10, 2021

Add security mechanism info to Kafka bindings for server objects asyncapi/spec#466

Closed

lbroudoux mentioned this pull request Mar 29, 2021

Implement channel name regex configuration Apicurio/apicurio-studio#1458

Merged

edeesis mentioned this pull request Jan 24, 2024

Custom SASL Mechanisms for Kafka #236

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add essential Kafka client security parameters to bindings #56

docs: add essential Kafka client security parameters to bindings #56

dalelane commented Feb 24, 2021

smoya commented Feb 24, 2021 •

edited

Loading

dalelane commented Feb 24, 2021

smoya commented Feb 24, 2021

fmvilas Feb 24, 2021

dalelane Feb 24, 2021

fmvilas Feb 24, 2021

fmvilas Feb 24, 2021

fmvilas Feb 24, 2021

dalelane Feb 24, 2021

fmvilas Feb 24, 2021

dalelane Feb 24, 2021

fmvilas commented Feb 24, 2021

nictownsend commented Feb 24, 2021

smoya commented Feb 25, 2021

dalelane commented Feb 25, 2021

dalelane commented Feb 25, 2021


		Field Name \| Type \| Description \| Applicability [default] \| Constraints

docs: add essential Kafka client security parameters to bindings #56

docs: add essential Kafka client security parameters to bindings #56

Conversation

dalelane commented Feb 24, 2021

smoya commented Feb 24, 2021 • edited Loading

dalelane commented Feb 24, 2021

smoya commented Feb 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmvilas commented Feb 24, 2021

nictownsend commented Feb 24, 2021

smoya commented Feb 25, 2021

dalelane commented Feb 25, 2021

dalelane commented Feb 25, 2021

smoya commented Feb 24, 2021 •

edited

Loading