Schema Registry
Schema Registry is a service that manages and enforces schemas for data in Kafka. It ensures data consistency by validating schemas during serialization and deserialization, supports schema versioning, and enforces compatibility rules to prevent breaking changes. This enables reliable data exchange across producer and consumer components in an ecosystem. When handling a large amount of data, issues such as high latency, scalability challenges, high CPU and memory usage, schema validation failures, inconsistent schema versions, etc., might occur.
To address these challenges, the Pulse observability platform provides the following capabilities or metrics:
- Monitors the overall health and performance of Schema Registry by providing the number of requests made to Schema Registry and the time taken to process requests.
- Provides the percentage of successful and failed requests processed by Schema Registry.
- Provides the number of failed requests due to various errors, such as schema validation failures or compatibility issues.
- Provides the time taken to retrieve schema information by its ID from Schema Registry, register a new version, and process delete requests for schemas in Schema Registry.
- The amount of memory utilized by Java Virtual Machine (JVM) running the Schema Registry.
You can use these details shown on the Pulse UI to plan capacity based on the current usage pattern, optimize schema storage, and ensure that the Schema Registry is scaled appropriately to handle the data load.
The Schema Registry service is an API-based service that can be used for Kafka, NiFi, etc.
For details about setting up Schema Registry, see Schema Registry.
Schema Registry on the Pulse UI
You can navigate to the Pulse UI > Menu on the top left pane and select Schema Registry to access its UI page.
Monitoring the Schema Registry Charts
The following charts provide the Schema Registry activity details.
On any chart, you can perform the following activities.
- Refresh a chart
- Enlarge a chart
- Download a chart
| Chart | Description |
|---|---|
| Success Response Rate | This chart shows the count of successful responses from Kafka Cruise Control. The graph represents the success response rate for different HTTP status code categories over time.
|
| Failure Response Rate | This chart shows the count of failed responses from Kafka Cruise Control. The graph represents the failure response rate for different HTTP status code categories over time.
|
| Schema Registry Error Count | This chart shows the number of errors encountered by the Schema Registry. |
| Get Schema Info By ID Latency | This chart shows the performance metrics (latency) of the Get Schema Info By ID operation in a Schema Registry, measured at the 50th percentile (median) and 99th percentile. |
| Get Schema Info By ID Throughput | This chart shows the number of Get Schema Info By ID requests the Schema Registry can handle per unit of time. |
| Get Schema Info By ID Latency (Confluent) | This chart shows the performance metrics (latency) of the Get Schema Info By ID operation in a Schema Registry, measured at the 50th percentile (median) and 99th percentile (compatible with Confluent API). |
| Get Schema Info ByI D Throughput (Confluent) | This chart shows the number of Get Schema Info By ID requests the Schema Registry can handle per unit of time (compatible with Confluent API). |
| RegisterSchemaVersion Latency | This chart shows the 50th and 99th percentile latency values in registering a new schema version in a Schema Registry. |
| RegisterSchemaVersion Throughput | This shows the number of schema registration requests processed by the Schema Registry per unit of time. |
| RegisterSchemaVersion Latency (Confluent) | This shows the 50th and 99th percentile latency values in registering a new schema version in a Schema Registry (compatible with Confluent API). |
| RegisterSchemaVersion Throughput (Confluent) | This shows the number of schema registration requests processed by the Schema Registry per unit of time (compatible with Confluent API). |
| DeleteRequests Latency | This chart shows the time taken to process delete requests, measured at the 50th and 99th percentiles. |
| DeleteRequests Throughput | This chart shows the number of successful delete operations processed per unit of time. |
| Schema Registry Admin Threads | This chart shows the number of threads dedicated to the Schema Registry administration tasks. |
| Schema Registry Server Threads | The chart shows the number of threads dedicated to handling the Schema Registry server requests. |
| JVM Memory Usage | This chart shows the amount of JVM memory used by processes in the cluster. |
| JVM GC Count | This chart shows the number of garbage collection events performed by the JVM. |
| JVM GC Time | This chart shows the total time spent on garbage collection by the JVM. |
Monitoring the Schema Registry Alerts
You can enable the following Schema Registry alerts on the Alerts page. This helps you debug and fix any failures encountered in the pipeline.
To access the Alerts page, log on to the Pulse UI and navigate to the top left Menu > Alerts. On the Alerts page, either search or select the Schema Registry service from the filter list. The list of Schema Registry alerts appears on the screen.
| Alert | Description | Configuration |
|---|---|---|
SCHEMA_REGISTRY ADMIN_ENDPOINT_CHECK | This alert checks whether the registry admin is active or not. | Severity: "High", Execution Interval: "30" |
SCHEMA_REGISTRY SERVER_ENDPOINT_CHECK | This alert checks whether the registry server is active or not. | Severity: "Critical", Execution Interval: "60" |
SCHEMA_REGISTRY_HTTP_CLIENT_ERROR | This alert checks the schema registry client errors (4xx http responses). | Severity: "Medium", Execution Interval: "30" |
SCHEMA_REGISTRY HTTP_SERVER_ERROR | This alert checks the schema registry internal server errors (5xx http responses). | Severity: "Medium", Execution Interval: "30" |
SCHEMA_REGISTRY ERRORS | This alert checks the schema registry server errors. | Severity: "Medium", Execution Interval: "30" |
SCHEMA_REGISTRY NO_DATA_ALERT`` | This alert is raised when the schema registry data is not pushed. | Severity: "Medium", Execution Interval: "120" |
For the complete list of stock and predefined alerts, see Stock and Predefined Alerts.
For details about customizing and creating alerts, see Creating Alerts.
Monitoring the Schema Registry Logs and Service details
You can check the Logs and Service details by navigating to the respective UI pages.
- To check the Logs of Schema Registry, navigate to Menu > Logs. On the Logs page, select registry_server under Services. The Information, Warning, and Error logs are shown to monitor the health of the Schema Registry and these details help you in debugging. For more details, see Logs.
- To check the current status and the other metrics of the Schema Registry service, navigate to Menu > Nodes. On the Nodes page, select SchemaRegistry under Services. The service or application metrics such as status, load, CPU and Disk usage, etc. are displayed on the Nodes page. For more details, see Nodes.