Configure HDP Spark

This page helps you enable Spark metrics via the Graphite sink, allowing Pulse to capture Spark application and JVM metrics.

Configure Spark Metrics Sink

In the Ambari UI:

  1. Navigate to Spark > Configs > Advanced spark2-metrics-properties.

  2. Add the following properties:

# Graphite sink class *.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink # Location of your graphite instance *.sink.graphite.host=localhost *.sink.graphite.port=12003 *.sink.graphite.protocol=tcp *.sink.graphite.prefix=spark.metrics *.sink.graphite.period=20 master.source.jvm.class=org.apache.spark.metrics.source.JvmSource worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
Info

This change requires Pulse Node Agent running on all spark clients.

Steps for Spark Hook Update

  1. In the Ambari UI, navigate to Spark > Configs > Custom spark-defaults.

  2. Add or update the following properties:

spark.ad.connector.context yarn;<ClusterName>;admin;test spark.ad.nats.server.host.list http://<nats1>:19009,http://<nats2>:19009 spark.extraListeners io.acceldata.sparkstats.AdSparkListener spark.ad.events.url <pulse_ip>:19009

  1. Save the configuration changes.

  2. Remove the existing Spark hook JARs from the $SPARK_HOME/jars directory on all Spark client nodes.

  3. Copy the compatible Spark hook JARs (version 2.1.2) to the $SPARK_HOME/jars directory on all Spark client nodes. For the supported hook version on each Pulse release, see Supported Hook Versions.

  4. Restart the affected Spark services.

  5. Submit a test Spark job and verify that Spark events and metrics are successfully reported to Pulse.

Additional Notes

  • On edge nodes not managed by Ambari, manually append the above properties to:

/etc/spark2/conf/metrics.properties

  • Ensure the following properties are enabled in spark-defaults.conf for all Spark jobs:

spark.eventLog.enabled=true spark.eventLog.dir=hdfs:///spark2-history/

  • Apply the same properties in managed configurations for applications running on Spark 1.x and 3.x.

Result

  • Spark sends metrics (including JVM metrics) to Graphite for Pulse ingestion.


Next to read:
Configure CDP Cluster Services for Pulse

For additional help, contact www.acceldata.force.com OR call our service desk +1 844 9433282

Copyright © 2026

On This Page
Configure HDP SparkConfigure Spark Metrics SinkSteps for Spark Hook UpdateAdditional NotesResult