Kafka

Acceldata ODP introduces seamless integration with various data systems, enhancing data flow and compatibility across diverse platforms.

Enable Kerberos - Ambari

Enabling Kerberos for the entire ODP cluster includes enabling Kafka. For more information, see Enabling Kerberos in an ODP Cluster.

Enable Kafka SSL

Gather the CA-signed truststore and keystore certificates along with details to enable SSL for the Kafka Service.

If the CA certificates are not available, generate self-signed certificates by performing the following instructions:

1. Replicate the provided script on all the nodes where Kafka services are running:

cat Kafka-ssl.sh

password="<password>"

mkdir -p /opt/security/pki/

cd /opt/security/pki/ || exit

​

keytool -genkey -alias "$(hostname)" -keyalg RSA -keysize 2048 -dname "CN=$(hostname -f),OU=SU,O=ACCELO,L=BNG,ST=KN,C=IN" -keypass "$password" -keystore keystore.jks -storepass "$password"

​

keytool -export -alias "$(hostname)" -keystore keystore.jks -file "$(hostname).crt" -storepass "$password"

​

yes | keytool -import -file "$(hostname).crt" -keystore truststore.jks -alias "$(hostname)-trust" -storepass "$password"

2. Execute it on all the Kafka nodes.

sh Kafka-ssl.sh

3. Copy the $(hostname).crt file to all other Kafka nodes reciprocally. Finally, import the certificate to the truststore on each node.

This will generate the certificates at /opt/security/pki/.

With the above generated certificates (or CA certificates), you must have details regarding the SSL certificates.

Configure Kerberos, SSL and Ranger to Kafka

Once Kerberos and SSL details are obtained as described in the preceding sections, proceed to configure them for the Kafka service.

listeners=SASL_SSL://localhost:6668,SASL_PLAINTEXT://localhost:6667

authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer

sasl.enabled.mechanisms=GSSAPI

sasl.mechanism.inter.broker.protocol=GSSAPI

security.inter.broker.protocol=SASL_PLAINTEXT

ssl.keystore.location=/opt/security/pki/keystore.jks

ssl.keystore.password=<password>

ssl.truststore.location=/opt/security/pki/truststore.jks

ssl.truststore.password=<password>

ssl.key.password=<password>

ssl.client.auth=none

Apart from the above, all remaining configurations are to be set to default. Kafka is now enabled with Ranger, Kerberos, and SSL.

Basic Kafka Operations

NOTE The below sample operations are tested for a minimum of 3-nodes cluster.

Since the cluster is Kerberos and SSL enabled, use client-ssl.properties file to perform basic operations.

cat client-ssl.properties

security.protocol = SASL_SSL

ssl.truststore.location = /opt/security/pki/truststore.jks

ssl.truststore.password = <password>

Create a Topic

To create a Kafka topic, perform the following:

bin/kafka-topics.sh  \

    --create \

    --topic conn-test-0104-01 \

    --bootstrap-server <kafka-broker-host>:6668 \

    --partitions 3 \

    --replication-factor 3 \

    --command-config client-ssl.properties

Describe a Topic

To describe a Kafka topic, perform the following:

./bin/kafka-topics.sh \

        --describe \

        --topic conn-test-0104-01 \

        --bootstrap-server <kafka-broker-host>:6668 \

         --command-config client-ssl.properties

Kafka Console Producer

To produce sample data from Kafka, perform the following:

/usr/odp/current/kafka-broker/bin/kafka-console-producer.sh \

        --topic conn-test-0104-02 \

        --bootstrap-server <kafka-broker-host>:6668 \

        --producer.config client-ssl.properties

Kafka Console Consumer

To consume sample data from Kafka, perform the following:

/usr/odp/current/kafka-broker/bin/kafka-console-consumer.sh  

                 --bootstrap-server basa1-cent7.acceldata.ce:6668 \

                 --topic conn-test-0104-01 \

                 --consumer.config client-ssl.properties \

                 --from-beginning