feat: add SASL security scheme types for use by Kafka specs #502

dalelane · 2021-02-25T13:15:42Z

As discussed in asyncapi/bindings#56 this
adds additional security scheme types. The motivation for adding
them is to enable description of secured Kafka clusters, however
the security protocols and mechanisms being added are not unique
to Kafka, so this commit adds them as generic security schemes so
they could be used by other protocols as well.

Fixes #466

Signed-off-by: Dale Lane [email protected]

github-actions

Welcome to AsyncAPI. Thanks a lot for creating your first pull request. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.

Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

nictownsend · 2021-02-25T17:14:56Z

Originally posted by @dalelane in asyncapi/bindings#56 (comment)

AsyncAPI: server `protocol`	AsyncAPI: security scheme `type`	meaning encryption?	meaning auth?	maps to Kafka client parameter `sasl.mechanism`	maps to Kafka client parameter `security.protocol`
`kafka`		no	no	unset	`PLAINTEXT`
`kafka`	`plain` ^*	no	yes, using SASL/PLAIN	`PLAIN`	`SASL_PLAINTEXT`
`kafka`	`scramSha256` ^*	no	yes, using SASL/SCRAM	`SCRAM-SHA-256`	`SASL_PLAINTEXT`
`kafka`	`scramSha512` ^*	no	yes, using SASL/SCRAM	`SCRAM-SHA-512`	`SASL_PLAINTEXT`
`kafka`	`oauth2`	no	yes, using OAuth	`OAUTHBEARER`	`SASL_PLAINTEXT`
`kafka`	`gssapi` ^*	no	yes, using GSSAPI	`GSSAPI`	`SASL_PLAINTEXT`
`kafka-secure`		yes	no	unset	`SSL`
`kafka-secure`	`plain` ^*	yes	yes, using SASL/PLAIN	`PLAIN`	`SASL_SSL`
`kafka-secure`	`scramSha256` ^*	yes	yes, using SASL/SCRAM	`SCRAM-SHA-256`	`SASL_SSL`
`kafka-secure`	`scramSha512` ^*	yes	yes, using SASL/SCRAM	`SCRAM-SHA-512`	`SASL_SSL`
`kafka-secure`	`oauth2`	yes	yes, using OAuth	`OAUTHBEARER`	`SASL_SSL`
`kafka-secure`	`gssapi` ^*	yes	yes, using GSSAPI	`GSSAPI`	`SASL_SSL`
`kafka-secure`	`X509`	yes	yes, using mutual TLS	unset	`SSL`

^* - a new security scheme type to be added

nictownsend · 2021-02-25T17:15:42Z

This is all working on the assumption that kafka/kafka-secure is equivalent to http/https - https://asyncapi.slack.com/archives/C34F2JV0U/p1614255214067200 @fmvilas for reference

If it's not, then there will need to be more to achieve the table above.

smoya · 2021-03-02T15:12:16Z

versions/2.0.0/asyncapi.md

@@ -1961,7 +1961,7 @@ Defines a security scheme that can be used by the operations. Supported schemes
 ##### Fixed Fields
 Field Name | Type | Applies To | Description
 ---|:---:|---|---
-<a name="securitySchemeObjectType"></a>type | `string` | Any | **REQUIRED**. The type of the security scheme. Valid values are `"userPassword"`, `"apiKey"`, `"X509"`, `"symmetricEncryption"`, `"asymmetricEncryption"`, `"httpApiKey"`, `"http"`, `oauth2`, and `openIdConnect`.
+<a name="securitySchemeObjectType"></a>type | `string` | Any | **REQUIRED**. The type of the security scheme. Valid values are `"userPassword"`, `"apiKey"`, `"X509"`, `"symmetricEncryption"`, `"asymmetricEncryption"`, `"httpApiKey"`, `"http"`, `"oauth2"`, and `"openIdConnect"`, `"plain"`, `"scramSha256"`, `"scramSha512"`, `"gssapi"`.


Wouldn't plain be covered already by the userPassword?

Would it make sense to reuse the userPassword schema, adding an optional field for the encryption? Sorry for the back and forth 😅

I think that depends whether we want userPassword and X509 to be reusable across protocol.

i.e it's inferred (or we state) - that for kafka, userPassword = SASL plain, X509 = mutual TLS, scramShaX = SASL scram etc

I guess it becomes a challenge if kafka (or other protocols) add support for clashing ideas (e.g SASL x509 + mutual tls)

adding an optional field for the encryption

Can we assume that protocol kafka-secure means encrypted and kafka is plain?

i.e it's inferred (or we state) - that for kafka, userPassword = SASL plain, X509 = mutual TLS, scramShaX = SASL scram etc

Perhaps it makes sense to add the concept of aliases that at the end are just reusing the same JSON Schema types? What I would like to avoid is redefining the same concept with a different name and maintain both at the same time.

Can we assume that protocol kafka-secure means encrypted and kafka is plain?

Yeah, that's a good point 👍 . What does it should mean -secure suffix here? I guess the question is: Do we talk about encryption or the protocol?

What does it should mean -secure suffix here? I guess the question is: Do we talk about encryption or the protocol?

If we compare http/https - the s represents secure protocol (via SSL encryption) - at least, I believe that's the case. If so, then that maps accordingly to kafka/kafka-secure - the secure means it's a secure version of the protocol?

If we compare http/https - the s represents secure protocol (via SSL encryption)

I agree with this. I think being consistent with the distinction between http and https would be best - so in the same way that using https means encryption and does not require authentication, I think kafka-secure should mean encryption but not require auth.

Would it make sense to reuse the userPassword schema, adding an optional field for the encryption?

I don't really mind.

The root requirement is that there at least three SASL mechanisms typically used for Kafka that could all be described as username/password-based (SASL/PLAIN, SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512) so we either need to describe them as different security scheme types, or make this SASL mechanism information a new additional attribute of the security scheme.

Either is totally fine with me, as long as we have a way to articulate the required config.

@fmvilas Any strong preferences?

Not really. Either way is fine for me. Like you said, as long as we have a way to articulate the config, everything is fine. Just let's make sure it's not too verbose and it's maintainable in the long-term.

sonarqubecloud · 2021-03-16T20:55:05Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

dalelane · 2021-03-16T20:57:10Z

Updated the PR to address merge conflicts.

I think all the comments above are addressed and this is ready for re-review now, please.

dalelane · 2021-04-20T20:01:23Z

@fmvilas @derberg Is there anything else needed for this one, please?

fmvilas · 2021-04-21T12:05:56Z

I think it's pretty much done on the spec side. The only missing part for me is to update the JSON Schema definition and the JS Parser.

Regarding the JSON Schema definition, create a new document called 2.1.0.json and add these new security schemes to the definition here. Just create a pull request, doesn't have to be merged.

Regarding the JS Parser, similar thing: create a PR and make the @asyncapi/specs (this is asyncapi-node above) point to the branch you created above. Again, PR doesn't have to be merged.

If you meet these two criteria, we'll move this to the next stage (Draft RFC 2).

If you're not familiar with JS, we're happy to help on that front.

As discussed in asyncapi/bindings#56 this adds additional security scheme types. The motivation for adding them is to enable description of secured Kafka clusters, however the security protocols and mechanisms being added are not unique to Kafka, so this commit adds them as generic security schemes so they could be used by other protocols as well. Contributes to: #466 Signed-off-by: Dale Lane <[email protected]>

Moved the "and" to the end of the list. Signed-off-by: Dale Lane <[email protected]>

fmvilas · 2021-04-23T20:01:53Z

Moving this to the next stage 🎉 It's now a Draft, pending testing it on the JS parser.

derberg · 2021-04-26T08:05:15Z

In my opinion, we should also get examples of new schemes like in case of others from the list https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md#security-scheme-object-example, also for me it is not clear what is the structure of information that one has to provide when using scramSha512 for example.

above the table that is modified in this PR is the list of supported schemes, would be good to add the new ones there and even link them to some vendor-neutral links (if available) that explain more.

Thoughts?

dalelane · 2021-04-26T08:13:46Z

In my opinion, we should also get examples of new schemes like in case of others from the list https://github.com/asyncapi/spec/blob/master/spec/asyncapi.md#security-scheme-object-example

Sure, I can add that.

it is not clear what is the structure of information that one has to provide when using scramSha512

For now, there isn't yet extra information - the crucial info is the type itself. Knowing that the SASL mechanism is SASL/SCRAM-SHA-512 is what you need to know to connect to the cluster (other than the actual credentials themselves)

above the table that is modified in this PR is the list of supported schemes, would be good to add the new ones there and even link them to some vendor-neutral links (if available) that explain more.

Good idea, I can add that

Contributes to: #466 Signed-off-by: Dale Lane <[email protected]>

sonarqubecloud · 2021-04-26T09:07:15Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

derberg · 2021-05-14T10:03:36Z

what do you think about extending the streetlight example?

derberg · 2021-05-24T12:22:53Z

@dalelane can you have a look at my last comment?

derberg · 2021-05-26T08:35:41Z

@fmvilas I checked the streetlights example and I actually revert my request to @dalelane to extend it with SASL, as SASL is for Kafka and example is for MQTT. Problem is that this example, even though for mqtt, has kafka bindings. So the example is a bit messy and I would not like to block this PR because of it.

I had quick chat with @dalelane and he, as our Kafka ❤️ AsyncAPI expert, will work on another PR to fix the existing example and provide a separate Kafka example.

@fmvilas I think we are ready to go with these PRs related to SASL. The way to merge should be:

spec PR merge
asyncapi-node PR merge
wait for asyncapi-node release candidate for NPM
bump this candidate in PR in parser-js
parser-js PR merge

I think this is it 🤔

fmvilas · 2021-05-28T07:53:59Z

Yeah, agree. And since we all agree, I'm marking it as 🏁 RFC Stage 3 (Accepted) 🎉

Congrats, @dalelane! It's the first accepted RFC after v2.0.0.

asyncapi-bot · 2021-05-28T08:04:30Z

🎉 This PR is included in version 2.1.0-2021-06-release.2 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

dalelane · 2021-05-31T21:01:03Z

@derberg FYI - I've had a go at preparing a new version of the Streetlight example in #548

Main motivation of this was to provide a place to demonstrate the new Kafka-specific security schemes introduced in AsyncAPI 2.1.0. It was also a chance to remove the Kafka bindings from the otherwise-MQTT sample. The need for this was discussed in #502 Contributes to: #466 Signed-off-by: Dale Lane <[email protected]>

asyncapi-bot · 2021-06-29T08:21:20Z

🎉 This PR is included in version 2.2.0-2021-06-release.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

github-actions bot reviewed Feb 25, 2021

View reviewed changes

dalelane changed the title ~~Add SASL security scheme types for use by Kafka specs~~ feat: Add SASL security scheme types for use by Kafka specs Feb 25, 2021

dalelane changed the title ~~feat: Add SASL security scheme types for use by Kafka specs~~ feat: add SASL security scheme types for use by Kafka specs Feb 25, 2021

dalelane mentioned this pull request Feb 25, 2021

docs: add essential Kafka client security parameters to bindings asyncapi/bindings#56

Closed

dalelane mentioned this pull request Feb 26, 2021

AsyncAPI open meeting, Tuesday Mar 2 2021 4PM UTC #504

Closed

smoya reviewed Mar 2, 2021

View reviewed changes

fmvilas added the 💡 Proposal (RFC 1) RFC Stage 1 (See CONTRIBUTING.md) label Apr 21, 2021

fmvilas added 📣 RFC document PR creates or changes document inside "rfc" folder and removed 📣 RFC document PR creates or changes document inside "rfc" folder labels Apr 21, 2021

dalelane added 2 commits April 22, 2021 21:05

chore: update for grammar

a05a1e4

Moved the "and" to the end of the list. Signed-off-by: Dale Lane <[email protected]>

dalelane mentioned this pull request Apr 22, 2021

Add security mechanism info to Kafka bindings for server objects #466

Closed

fmvilas changed the base branch from master to 2021-06-release April 23, 2021 19:56

fmvilas added 📄 Draft (RFC 2) RFC Stage 2 (See CONTRIBUTING.md) and removed 💡 Proposal (RFC 1) RFC Stage 1 (See CONTRIBUTING.md) labels Apr 23, 2021

docs: additional SASL documentation

538f031

Contributes to: #466 Signed-off-by: Dale Lane <[email protected]>

dalelane mentioned this pull request May 10, 2021

Define a release cadence #513

Closed

fmvilas added this to the 2.1.0 Release milestone May 14, 2021

fmvilas added 🏁 Accepted (RFC 3) RFC Stage 3 (See CONTRIBUTING.md) and removed 📄 Draft (RFC 2) RFC Stage 2 (See CONTRIBUTING.md) labels May 28, 2021

fmvilas merged commit 8c8e211 into asyncapi:2021-06-release May 28, 2021

asyncapi-bot added the released on @2021-06-release label May 28, 2021

github-actions bot approved these changes May 28, 2021

View reviewed changes

dalelane mentioned this pull request May 31, 2021

docs: separate streetlight examples for Kafka and MQTT #548

Merged

dalelane deleted the security-scheme-for-kafka branch May 31, 2021 21:00

derberg mentioned this pull request Feb 10, 2022

Suggestion to extend CODEOWNERS with new contributor #720

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add SASL security scheme types for use by Kafka specs #502

feat: add SASL security scheme types for use by Kafka specs #502

dalelane commented Feb 25, 2021 •

edited by fmvilas

Loading

github-actions bot left a comment

nictownsend commented Feb 25, 2021 •

edited

Loading

nictownsend commented Feb 25, 2021

smoya Mar 2, 2021

smoya Mar 2, 2021 •

edited

Loading

nictownsend Mar 2, 2021

nictownsend Mar 2, 2021

nictownsend Mar 2, 2021 •

edited

Loading

smoya Mar 2, 2021

nictownsend Mar 2, 2021

dalelane Mar 11, 2021 •

edited

Loading

dalelane Mar 11, 2021

fmvilas Mar 11, 2021

sonarqubecloud bot commented Mar 16, 2021

dalelane commented Mar 16, 2021

dalelane commented Apr 20, 2021

fmvilas commented Apr 21, 2021

fmvilas commented Apr 23, 2021

derberg commented Apr 26, 2021

dalelane commented Apr 26, 2021

sonarqubecloud bot commented Apr 26, 2021

derberg commented May 14, 2021

derberg commented May 24, 2021

derberg commented May 26, 2021 •

edited

Loading

fmvilas commented May 28, 2021

asyncapi-bot commented May 28, 2021

dalelane commented May 31, 2021

asyncapi-bot commented Jun 29, 2021

feat: add SASL security scheme types for use by Kafka specs #502

feat: add SASL security scheme types for use by Kafka specs #502

Conversation

dalelane commented Feb 25, 2021 • edited by fmvilas Loading

github-actions bot left a comment

Choose a reason for hiding this comment

nictownsend commented Feb 25, 2021 • edited Loading

nictownsend commented Feb 25, 2021

smoya Mar 2, 2021

Choose a reason for hiding this comment

smoya Mar 2, 2021 • edited Loading

Choose a reason for hiding this comment

nictownsend Mar 2, 2021

Choose a reason for hiding this comment

nictownsend Mar 2, 2021

Choose a reason for hiding this comment

nictownsend Mar 2, 2021 • edited Loading

Choose a reason for hiding this comment

smoya Mar 2, 2021

Choose a reason for hiding this comment

nictownsend Mar 2, 2021

Choose a reason for hiding this comment

dalelane Mar 11, 2021 • edited Loading

Choose a reason for hiding this comment

dalelane Mar 11, 2021

Choose a reason for hiding this comment

fmvilas Mar 11, 2021

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 16, 2021

dalelane commented Mar 16, 2021

dalelane commented Apr 20, 2021

fmvilas commented Apr 21, 2021

fmvilas commented Apr 23, 2021

derberg commented Apr 26, 2021

dalelane commented Apr 26, 2021

sonarqubecloud bot commented Apr 26, 2021

derberg commented May 14, 2021

derberg commented May 24, 2021

derberg commented May 26, 2021 • edited Loading

fmvilas commented May 28, 2021

asyncapi-bot commented May 28, 2021

dalelane commented May 31, 2021

asyncapi-bot commented Jun 29, 2021

dalelane commented Feb 25, 2021 •

edited by fmvilas

Loading

nictownsend commented Feb 25, 2021 •

edited

Loading

smoya Mar 2, 2021 •

edited

Loading

nictownsend Mar 2, 2021 •

edited

Loading

dalelane Mar 11, 2021 •

edited

Loading

derberg commented May 26, 2021 •

edited

Loading