Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOC-2085 fix(dataloading): improve doc for Load from External Kafka (3.9?-4.1) #491

Open
wants to merge 6 commits into
base: 4.1
Choose a base branch
from
+32 −55
Open

DOC-2085 fix(dataloading): improve doc for Load from External Kafka (3.9?-4.1) #491

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ea27181
Revising documentation about data-loading.
wuqingjuntg Aug 6, 2024
adb8200
To make it simple, let's support SSL only, and not SASL(including SAS…
wuqingjuntg Aug 7, 2024
6de9366
sql -> gsql
wuqingjuntg Aug 12, 2024
a52c0ce
Addressing comments.
wuqingjuntg Aug 12, 2024
8142602
Merge branch '4.1.o' into DOC-2085
wuqingjuntg Aug 19, 2024
3101fbb
Resolving code review comments.
wuqingjuntg Aug 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 6 additions & 37 deletions modules/data-loading/examples/config-avro
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
connector.class=org.apache.kafka.connect.mirror.MirrorSourceConnector
source.cluster.alias=hello
target.cluster.alias=world
source.cluster.bootstrap.servers=source.kafka.server:9092
target.cluster.bootstrap.servers=localhost:30002
source.cluster.bootstrap.servers=<source.broker1:port,source.broker2:port,...>
target.cluster.bootstrap.servers=<local.broker1:port,local.broker2:port,...>
source->target.enabled=true
topics=avro-without-registry-topic
replication.factor=1
Expand All @@ -18,41 +18,10 @@ emit.heartbeats.interval.seconds=5
world.scheduled.rebalance.max.delay.ms=35000
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
header.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=com.tigergraph.kafka.connect.converters.TigerGraphAvroConverterWithoutSchemaRegistry

producer.security.protocol=SASL_SSL
producer.sasl.mechanism=GSSAPI
producer.sasl.kerberos.service.name=kafka
producer.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\"/path/to/kafka-producer.keytab\" principal=\"[email protected]\";
producer.ssl.endpoint.identification.algorithm=
producer.ssl.keystore.location=/path/to/client.keystore.jks
producer.ssl.keystore.password=******
producer.ssl.key.password=******
producer.ssl.truststore.location=/path/to/client.truststore.jks
producer.ssl.truststore.password=******

consumer.security.protocol=SASL_SSL
consumer.sasl.mechanism=GSSAPI
consumer.sasl.kerberos.service.name=kafka
consumer.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\"/path/to/kafka-consumer.keytab\" principal=\"[email protected]\";
consumer.ssl.endpoint.identification.algorithm=
consumer.ssl.keystore.location=/path/to/client.keystore.jks
consumer.ssl.keystore.password=******
consumer.ssl.key.password=******
consumer.ssl.truststore.location=/path/to/client.truststore.jks
consumer.ssl.truststore.password=******

source.admin.security.protocol=SASL_SSL
source.admin.sasl.mechanism=GSSAPI
source.admin.sasl.kerberos.service.name=kafka
source.admin.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\"/path/to/kafka-admin.keytab\" principal=\"[email protected]\";
source.admin.ssl.endpoint.identification.algorithm=
source.admin.ssl.keystore.location=/path/to/client.keystore.jks
source.admin.ssl.keystore.password=******
source.admin.ssl.key.password=******
source.admin.ssl.truststore.location=/path/to/client.truststore.jks
source.admin.ssl.truststore.password=******
transforms=TigerGraphAvroTransform
transforms.TigerGraphAvroTransform.type=com.tigergraph.kafka.connect.transformations.TigergraphAvroWithoutSchemaRegistryTransformation
transforms.TigerGraphAvroTransform.errors.tolerance=none

[connector_1]
name=avro-test-without-registry
tasks.max=10
tasks.max=10
3 changes: 2 additions & 1 deletion modules/data-loading/partials/kafka/kafka-data-source-details.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ If the source cluster is configured for SSL or SASL protocols, you need to provi
* If the source cluster uses SASL *and* SSL, you need to upload the keytab of each Kerberos principal, as well as the key store and truststore to every node of your TigerGraph cluster.
Each file must be at the same absolute path on all nodes.

The following configurations are required for admin, producer and consumer. To supply the configuration for the corresponding component, replace `<prefix>` with `source.admin`, `producer`, or `consumer`.
The following are generic configurations required for admin, producer and consumer. To supply the configuration for the corresponding component, replace `<prefix>` with `source.admin`, `producer`, or `consumer`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you tried it?

  • Are source.admin, producer, or consumer all required?
  • Why admin requires source prefix but others are not?
  • How about using source.cluster?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should have mentioned this earlier:

  • We already have a separate instructions on how to set up security (SSL), so user can just follow that instruction to set up SSL, by gadmin enabling a couple of flags, setting certificates etc.;
  • This example is simply about avro/mm2, so let's try to make this example as simple as possible;
  • This document was provided when SSL is not supported; now given SSL is already supported, we can just recommend user to use SSL if they are concerned about security;
  • All these settings are covered by our current regress test, e.g., cqrs_cluster 101.
  • I would recommend to remove all the wording including SASL from the document. We have a plan to support SASL, but we haven't implemented/supported it yet.

Let me know your thoughts.
Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these settings are covered by our current regress test, e.g., cqrs_cluster 101.

  • cqrs_cluster 101 is for CRR with SSL, not loading from external Kafka. The SSL setting should be different because the CRR SSL setting is auto-generated.
  • In the auto-generated CRR config you wrote, it is using source.cluster.ssl instead of source.admin, producer or consumer

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • No. cqrs_cluster 101 includes two test cases: test1.sh and test2.sh. test1.sh is for generic mm2; test2.sh is for crr.
  • When I designed SSL for MM2/CRR, connector_manager was used to set up ssl configuration settings to improve the usability.
  • You seemed to ask similar question when you reviewed the SSL PR, here is one of the comments: https://github.com/tigergraph/cqrs/pull/3001/files#r1432225187 . Basically, Kafka(KafkaConn) maintains precedence of SSL settings: (source or target).cluster.ssl > admin/producer/consumer.ssl etc.. In the design, I just made it simple: all admin/producer/consumer share the same SSL settings to make things simple.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically, Kafka(KafkaConn) maintains precedence of SSL settings: (source or target).cluster.ssl > admin/producer/consumer.ssl etc.

  • Have we explained this to the user? That's why I'm asking to improve the doc.
  • Even if we use admin, consumer and producer separately, there's still some mistakes in the doc: why source.admin but not no source. for consumer and producer?

For example, to specify `GSSAPI` as the SASL mechanism for consumer, include `"consumer.sasl.mecahnism": "GSSAPI"` in the data source configuration.
But if the source cluster uses SSL, please skip the generic configurations below and follow this documentation to setup connector xref:tigergraph-server:data-loading:kafka-ssl-security-guide.adoc[]

[%header,cols="1,2"]
|===
Expand Down
2 changes: 1 addition & 1 deletion modules/data-loading/partials/kafka/kafka-example-loading-job.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The following is an example loading job from and external Kafka cluster.

[source,php,linenums]
[source,sql,linenums]
pingxieTG marked this conversation as resolved.
Show resolved Hide resolved
pingxieTG marked this conversation as resolved.
Show resolved Hide resolved
.Example loading job for BigQuery
pingxieTG marked this conversation as resolved.
Show resolved Hide resolved
----
USE GRAPH ldbc_snb
Expand Down