Configure CDP Spark

This page describes configuring Spark metrics in CM to enable JMX monitoring and Graphite reporting for Spark and Spark2 clients.

Steps for Configuration

In the CM UI, navigate to Service > Configuration tab.
In the search bar, type the following, then update the parameters with the appropriate values.

Spark Client Advanced Configuration Snippet (Safety Valve) → maps to spark-conf/spark-defaults.conf.
Spark2 Client Advanced Configuration Snippet (Safety Valve) → maps to spark2-conf/spark-defaults.conf.

spark.metrics.conf.*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink spark.metrics.conf.*.sink.graphite.host=localhost spark.metrics.conf.*.sink.graphite.port=12003 spark.metrics.conf.*.sink.graphite.protocol=tcp spark.metrics.conf.*.sink.graphite.prefix=spark.metrics spark.metrics.conf.*.sink.graphite.period=20 spark.metrics.conf.*.source.jvm.class=org.apache.spark.metrics.source.JvmSource spark.ad.connector.context yarn;<ClusterName>;admin;test spark.ad.nats.server.host.list http://<nats1>:19009,http://<nats2>:19009 spark.extraListeners io.acceldata.sparkstats.AdSparkListener spark.ad.events.url <pulse_ip>:19009

Result

Spark jobs will send JVM and application metrics to Graphite, enabling Pulse to collect, visualize, and analyze Spark performance.

Last updated on

Was this page helpful?