Kafka Connect

Kafka Connect is a service within the Apache Kafka ecosystem designed to streamline the integration of Kafka with other systems. It simplifies the process of moving large amounts of data between Kafka and various data sources or sinks.

Kafka Connect configurations can quickly become complex when managing multiple connectors, tasks, and transformations, leading to skewed workers, rebalance failures, and connection problems.

To address these challenges efficiently, the Pulse observability platform monitors the entire data pipeline between Kafka and other systems, providing detailed insights into the nature and location of failures. This allows for quick corrective actions to resolve issues in the data pipeline.

Acceldata Pulse monitors the Kafka Connect cluster Worker nodes, Connectors, Tasks, Converter, Dead Letter Queues, etc., and shows their metrics, errors, and failures on the Kafka Connect Dashboard page.

For details about setting up Kafka Connect and the supported connector integration, see Kafka Connect.

Kafka Connect on the Pulse UI

To access Kafka Connect, log on to the Pulse UI and navigate to the top left Menu > Kafka > Connect.

The Pulse observability platform provides the following capabilities:

Monitoring the Kafka Connect Workers

There can be multiple Workers in the Kafka Connect cluster, and you can monitor the health of all Workers by enabling the Kafka Connect Worker Endpoint Check alert on the Alerts page.

Pulse observes the health of all Workers and sends an Alert when any Worker is unreachable.

Follow the below steps to enable the Endpoint Check Alert:

Log on to the Pulse UI and navigate to Alerts.
On the Alerts page, select Kafka in Filters.
On the Kafka table, click Enabled for KAFKA_CONNECT_WORKER_ENDPOINT_CHECK.

Monitoring the Kafka Connect Logs and Service details

You can check the Kafka Connect Logs and Service details by navigating to the respective UI pages.

The Logs page provides details such as information, errors, and warnings. This helps you to debug Kafka Connect when there are any issues.

The Nodes page provides details such as the number of hosts where Kafka Connect is deployed, application status, CPU Usage in %, Disk Usage in %, etc.

To check the Logs of the Kafka Connect, navigate to Menu > Logs. On the Logs page, select kafka_connect under Services. The Information, Warning, and Error logs are shown to monitor the health of the Kafka Connect, and these details help you in debugging. For more details, see Logs.
To check the current status and the other details of the Kafka Connect service, navigate to Menu > Nodes. On the Nodes page, select KafkaConnect under Services. The service or application metrics, such as status, load, CPU and Disk usage, etc., are displayed on the Nodes page. For more details, see Nodes.

Monitoring the Kafka Connect Components

You can monitor the Kafka Connect service running on all Nodes in a cluster, including the details such as CPU usage in %.

Pulse provides the KPI information about the component process status.

The component details are displayed on the page. For more details about using the functionalities of the Nodes page, see Nodes.

The Kafka Connect Summary Panel and Charts

You can check the Kafka Connect charts to understand the metrics in detail.

Log on to the Pulse UI and navigate to Menu.
On Menu, select Kafka > Connect.

Summary Panel

The Summary Panel provides the following Kafka Connect cluster information and Connector Tasks details.

Metric	Description
Kafka Connect Cluster Info	Provides the total number of workers, connectors, tasks, running tasks, paused tasks, unassigned tasks, destroyed tasks, and failed tasks running on a cluster.
Connector Tasks	Provides the list of connectors running on a cluster along with the total number of tasks, running tasks, paused tasks, unassigned tasks, destroyed tasks, and failed tasks for each connector.

You can hover over the Kafka Connect Cluster Info box and perform the following activities:

Refresh the cluster information

You can hover over the Cluster Tasks box and perform the following activities:

Refresh the Connector Tasks details
Enlarge the Connector Tasks space
Download the Connector Tasks report

Charts

The Kafka Connect charts provide the following details.

On any chart, you can perform the following activities.

Refresh a chart
Enlarge a chart
Download a chart

On a chart, you can hover over the tip icon to know about the chart details.

Chart	Description
Source Connector Poll Records	The total number of records produced/polled (before transformation) by tasks belonging to the named source connector.
Source Connector Write Records	The number of records output written to Kafka for tasks belonging to the named source connector. This is after the transformations are applied and exclude any records filtered out by the transformations.
Sink Connector Read Records	The number of records read from Kafka by the tasks belonging to the named sink connector.
Sink Connector Send Records	The total number of records output from the transformations and sent/put to tasks belonging to the named sink connector.
Source Connector Active Records	The total number of records that have been produced by this task but not yet completely written to Kafka.
Sink Connector Active Records	The number of records that have been read from Kafka but not yet completely committed, flushed, or acknowledged by the sinks tasks.
Connecter Running Tasks	The number of Connector running tasks.
Total Errors Logged	The total number of errors logged.
Dead Letter Queue Produce Requests	The number of attempted writes to the dead letter queue.
Connector Failed Tasks	The number of failed tasks of the connector.
Total Records Skipped	The number of records skipped due to errors.
Dead Letter Produce Failures	The number of failed writes to the dead letter queues.
Worker Connector Startup Failure Percentage	The average percentage of worker connectors that failed to start.
Worker Task Startup Failure Percentage	The average percentage of task connectors that failed to start.
Worker JVM	The total memory usage of the Kafka Connect worker's JVM.
Worker Active Connections	The number of active worker connections currently.
Worker CPU Usage	The amount of Kafka Connect Worker CPU usage.
Worker CPU Usage Aggregate	The total CPU usage across all Kafka Connect workers in a cluster.
Admin Client Connection Count	The number of active connections managed by the Admin Client.
Worker Time Since Last Rebalance	The time duration since the last rebalance operation occurred.

Monitoring the Kafka Connect Alerts

You can enable the following Kafka Connect alert on the Alerts page. This helps you debug and fix any failures encountered in the pipeline.

Pulse observes the health of all Kafka Connect components and sends an Alert when any of them fail to perform as expected.

To access the Alerts page, log on to the Pulse UI and navigate to the top left Menu > Alerts. On the Alerts page, select the Kafka service from the filter list. The list of alerts, including Kafka Connect alerts, appears on the screen.

Alert	Description	Configuration
`KAFKA_CONNECT_WORKER_ENDPOINT_CHECK`	This alert checks whether the Kafka Connect worker is active or not.	Severity: "Critical", Execution Interval: "60"
`KAFKA_CONNECT_OFFSET_COMMIT_FAILURE`	This alert check commits failure to Kafka topic while data ingestion from an external system (in case of Source connector) or commits failure while reading data from Kafka and writing to an external system (in case of Sink connector).	Severity: "Critical", Execution Interval: "30"
`KAFKA_CONNECT_CONNECTOR_STARTUP_FAILURE`	This alert checks if the worker's connectors are failing to start.	Severity: "Critical", Execution Interval: "30"
`KAFKA_CONNECT_TASK_STARTUP_FAILURE`	This alert checks if the worker's tasks are failing to start.	Severity: "Critical", Execution Interval: "30"
`KAFKA_CONNECT_NO_DATA_ALERT`	This alert is raised when the Kafka Connect data is not pushed.	Severity: "Medium", Execution Interval: "120"

For the complete list of stock and predefined alerts, see Stock and Predefined Alerts.

For details about customizing and creating alerts, see Creating Alerts.

Last updated on

Was this page helpful?