Configure CDH Spark

This page describes configuring Spark metrics in CM to enable JMX monitoring and Graphite reporting for Spark and Spark2 clients.

Steps for Configuration

1. In the CM UI, navigate to Service > Configuration tab.
2. In the search bar, type the below and update the following parameters with values.

• Spark Client Advanced Configuration Snippet (Safety Valve) → maps to spark-conf/spark-defaults.conf.
• Spark2 Client Advanced Configuration Snippet (Safety Valve) → maps to spark2-conf/spark-defaults.conf.

spark.metrics.conf.*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink

spark.metrics.conf.*.sink.graphite.host=localhost

spark.metrics.conf.*.sink.graphite.port=12003

spark.metrics.conf.*.sink.graphite.protocol=tcp

spark.metrics.conf.*.sink.graphite.prefix=spark.metrics

spark.metrics.conf.*.sink.graphite.period=20

spark.metrics.conf.*.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Result

Spark jobs will send JVM and application metrics to Graphite, enabling Pulse to collect, visualize, and analyze Spark performance.