Configure ODP Spark 2 and 3

This page helps you enable Spark metrics via the Graphite sink, allowing Pulse to capture Spark application and JVM metrics.

Configure Spark Metrics Sink

In the Ambari UI:

1. Navigate to Spark > Configs > Advanced spark2-metrics-properties.
2. Update the following parameters:

xxxxxxxxxx

# Graphite sink class

*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink

# Location of your graphite instance

*.sink.graphite.host=localhost

*.sink.graphite.port=12003

*.sink.graphite.protocol=tcp

*.sink.graphite.prefix=spark.metrics

*.sink.graphite.period=20

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource

worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource

driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource

executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Steps for Spark Hook Update

1. In the Ambari UI, navigate to Spark > Configs > Custom spark-defaults.
2. Add or update the following properties:

xxxxxxxxxx

spark.ad.connector.context         yarn;<ClusterName>;admin;test

spark.ad.nats.server.host.list     http://<nats1>:19009,http://<nats2>:19009

spark.extraListeners               io.acceldata.sparkstats.AdSparkListener

spark.ad.events.url                <pulse_ip>:19009

3. Save the configuration changes.
4. Remove the existing Spark hook JARs from the $SPARK_HOME/jars directory on all Spark client nodes.
5. Copy the compatible Spark hook JARs (For example, version 2.1.2) to the $SPARK_HOME/jars directory on all Spark client nodes. For the supported hook version on each Pulse release, see Supported Hook Versions.
6. Restart the affected Spark services.
7. Submit a test Spark job and verify that Spark events and metrics are successfully reported to Pulse.

Result

• Spark 2.x/3.x sends metrics (including JVM metrics) to Graphite for Pulse ingestion.