Configure CDH Spark

This page describes configuring Spark metrics in CM to enable JMX monitoring and Graphite reporting for Spark and Spark2 clients.

Steps for Configuration

In the CM UI, navigate to Service > Configuration tab.
In the search bar, type the following, then update the parameters with the appropriate values.

Spark Client Advanced Configuration Snippet (Safety Valve) → maps to spark-conf/spark-defaults.conf.
Spark2 Client Advanced Configuration Snippet (Safety Valve) → maps to spark2-conf/spark-defaults.conf.

spark.metrics.conf.*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink spark.metrics.conf.*.sink.graphite.host=localhost spark.metrics.conf.*.sink.graphite.port=12003 spark.metrics.conf.*.sink.graphite.protocol=tcp spark.metrics.conf.*.sink.graphite.prefix=spark.metrics spark.metrics.conf.*.sink.graphite.period=20 spark.metrics.conf.*.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Info

For details on configuring other details for Spark, see Additional Notes.

Result

Spark jobs will send JVM and application metrics to Graphite, enabling Pulse to collect, visualize, and analyze Spark performance.

Last updated on

Was this page helpful?