DOC-2085 fix(dataloading): improve doc for Load from External Kafka (3.9?-4.1)

[wuqingjuntg]

No description provided.

[pingxieTG]

Did you tried it?

• Are source.admin, producer, or consumer all required?
• Why admin requires source prefix but others are not?
• How about using source.cluster?

[wuqingjuntg]

I should have mentioned this earlier:

• We already have a separate instructions on how to set up security (SSL), so user can just follow that instruction to set up SSL, by gadmin enabling a couple of flags, setting certificates etc.;
• This example is simply about avro/mm2, so let's try to make this example as simple as possible;
• This document was provided when SSL is not supported; now given SSL is already supported, we can just recommend user to use SSL if they are concerned about security;
• All these settings are covered by our current regress test, e.g., cqrs_cluster 101.
• I would recommend to remove all the wording including SASL from the document. We have a plan to support SASL, but we haven't implemented/supported it yet.

Let me know your thoughts.
Thanks!

[pingxieTG]

All these settings are covered by our current regress test, e.g., cqrs_cluster 101.

• cqrs_cluster 101 is for CRR with SSL, not loading from external Kafka. The SSL setting should be different because the CRR SSL setting is auto-generated.
• In the auto-generated CRR config you wrote, it is using source.cluster.ssl instead of source.admin, producer or consumer

[wuqingjuntg]

• No. cqrs_cluster 101 includes two test cases: test1.sh and test2.sh. test1.sh is for generic mm2; test2.sh is for crr.
• When I designed SSL for MM2/CRR, connector_manager was used to set up ssl configuration settings to improve the usability.
• You seemed to ask similar question when you reviewed the SSL PR, here is one of the comments: https://github.com/tigergraph/cqrs/pull/3001/files#r1432225187 . Basically, Kafka(KafkaConn) maintains precedence of SSL settings: (source or target).cluster.ssl > admin/producer/consumer.ssl etc.. In the design, I just made it simple: all admin/producer/consumer share the same SSL settings to make things simple.

[pingxieTG]

Basically, Kafka(KafkaConn) maintains precedence of SSL settings: (source or target).cluster.ssl > admin/producer/consumer.ssl etc.

• Have we explained this to the user? That's why I'm asking to improve the doc.
• Even if we use admin, consumer and producer separately, there's still some mistakes in the doc: why source.admin but not no source. for consumer and producer?

[pingxieTG]

Why remove them?

[pingxieTG]

https://graphsql.atlassian.net/browse/DOC-1972?focusedCommentId=152682 see comment for kafka-ssl-security-guide.adoc.

• SSL for internal Kafka
• SSL for CRR
• SSL for loading from external Kafka

They should belong to 3 different categories(under 3 different pages) instead of putting them together.

[wuqingjuntg]

• Let's clarify here. We're not going to cover SSL for CRR here right?
• For internal and external Kafka, I don't see major difference. The "internal" Kafka is set up for testing loading from regular Kafka cluster including external Kafka. Let me know if there's any understanding gap. We can have a short discussion in Tuesday's meeting and get consensus about this with the team. ( At the moment I still have a couple of JIRA tickets to handle, which also have high priorities).
• I am removing the SASL and the SASL/SSL parameters, because: without SSL, SASL is not secured. SASL is authentication method, but without encryption, the connection is not secured. So we should simply recommend the users to enable regular SSL to secure the connection between Kafka clusters. User doesn't need to use both SASL + SSL. SSL is enough. And we didn't officially support SASL yet.
• My suggestion is that, for all the configuration/settings in the DOC, we should have already supported the feature and also have test cases covering those configs.

cc @arunramasami , FYI. Any comments regarding this DOC change I am making?