Kafka 4 Connect

Set up Kafka Connect from Ambari

When you install Kafka, ensure you select the appropriate nodes for installing the Kafka broker and Connect services.

Set up Kafka Connect with Kerberos, SSL, and Ranger

Since Kafka is already configured with SSL, Kerberos, and Ranger, the underlying topics maintain these settings during Kafka Connect operations. The following are the configurations for the Advanced kafka3-connect-distributed section in Kafka3's CONFIGS.

bootstrap.servers=<broker1-host>:7001,<broker2-host>:7001,<brokern-host>:7001 config.storage.replication.factor=3 config.storage.topic=connect-configs group.id=connect-cluster key.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true offset.flush.interval.ms=10000 offset.storage.replication.factor=3 offset.storage.topic=connect-offsets plugin.path=/usr/odp/current/kafka3-broker/libs/* listeners=localhost:8084 rest.advertised.listener=HTTPS status.storage.replication.factor=3 status.storage.topic=connect-status sun.security.krb5.debug=true value.converter=org.apache.kafka.connect.json.JsonConverter value.converter.schemas.enable=true sasl.enabled.mechanisms=GSSAPI sasl.kerberos.service.name=kafka sasl.mechanism=GSSAPI sasl.mechanism.inter.broker.protocol=GSSAPI security.inter.broker.protocol=SASL_SSL security.protocol=SASL_SSL ssl.key.password=<password> ssl.keystore.location=/opt/security/pki/keystore.jks ssl.keystore.password=<password> ssl.truststore.location=/opt/security/pki/truststore.jks ssl.truststore.password=<password> java.security.auth.login.config=/usr/odp/current/kafka-broker/config/kafka_jaas.conf java.security.krb5.kdc=<KDC server hostname> java.security.krb5.realm=<KDC server realm name> javax.security.auth.useSubjectCredsOnly=true consumer.bootstrap.servers=<broker1-host>:7001,<broker2-host>:7001,<brokern-host>:7001 consumer.security.protocol=SASL_SSL consumer.ssl.truststore.location=/opt/security/pki/truststore.jks consumer.ssl.truststore.password=<password> producer.bootstrap.servers=<broker1-host>:7001,<broker2-host>:7001,<brokern-host>:7001 producer.security.protocol=SASL_SSL producer.ssl.truststore.location=/opt/security/pki/truststore.jks

After you configure the settings above, restart the Kafka service, including the broker and Connect. Once the restart completes, Kafka Connect operates in Kerberos and SSL mode.

Kafka connector use cases

Acceldata ODP Kafka Connect supports the following connectors, included in this release:

  1. Pubsub Connector

  2. MongoDB Connector

  3. JDBC Connector

  4. S3 Connector

  5. Amazon SQS Sink Connector

  6. Amazon Kinesis Firehose Connector

  7. camel-aws-sqs-sink

The following table describes the supported connector type and library version for each connector.

Connector name

Supported type

Library version

Pubsub

Source & Sink

1.3.1

MongoDB

Source & Sink

3.0.0

JDBC

Source & Sink

6.12.0

S3

Sink only

2.15.0

Kinesis

Sink only

0.0.8

camel-aws-sqs-sink

Sink only

4.18.0


  Last updated