Configure NATS Connection Timeout for Spark Hook
The Spark Hook supports configurable NATS connection timeouts and a circuit breaker mechanism to prevent NATS connectivity issues from delaying Spark application processing.
When NATS becomes unavailable, the Spark Hook limits connection retries and temporarily stops reconnecting after consecutive failures. This allows Spark applications to continue running without delays caused by repeated connection attempts.
Configuration
To configure the NATS connection settings on the Spark node:
- Log in to the node where Apache Spark is installed.
- Open the spark-defaults.conf file.
File path: ${SPARK_HOME}/conf/spark-defaults.conf
- Add or update the following properties:
spark.ad.nats.connection.timeout.sec=2
spark.ad.nats.circuit.max.failures=3
spark.ad.nats.circuit.cooldown.min=5
- Save the configuration file.
Configuration Parameters
Property | Default | Description |
spark.ad.nats.connection.timeout.sec | 2 | Specifies the NATS connection timeout in seconds. |
spark.ad.nats.circuit.max.failures | 3 | Specifies the number of consecutive connection failures before the circuit breaker opens. |
spark.ad.nats.circuit.cooldown.min | 5 | Specifies how long, in minutes, the Spark Hook waits before attempting to reconnect to NATS. |
Behavior During NATS Outages
- The Spark Hook attempts to reconnect to NATS using the configured timeout.
- After the configured number of consecutive failures, the circuit breaker opens.
- While the circuit breaker is open, the Spark Hook skips publishing events without waiting for a connection.
- After the cooldown period, the Spark Hook attempts to reconnect.
Note
Events generated while NATS is unavailable might not be published to Pulse. However, Spark application processing continues without repeated NATS connection delays.
Have a suggestion?