This page helps you enable Spark metrics via the Graphite sink, allowing Pulse to capture Spark application and JVM metrics.
Configure Spark Metrics Sink
In the Ambari UI:
- Navigate to Spark > Configs >
Advanced spark2-metrics-properties. - Add the following properties:
x
# Graphite sink class*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink# Location of your graphite instance*.sink.graphite.host=localhost*.sink.graphite.port=12003*.sink.graphite.protocol=tcp*.sink.graphite.prefix=spark.metrics*.sink.graphite.period=20master.source.jvm.class=org.apache.spark.metrics.source.JvmSourceworker.source.jvm.class=org.apache.spark.metrics.source.JvmSourcedriver.source.jvm.class=org.apache.spark.metrics.source.JvmSourceexecutor.source.jvm.class=org.apache.spark.metrics.source.JvmSourceThis change requires Pulse Node Agent running on all spark clients.
Additional Notes
- On edge nodes not managed by Ambari, manually append the above properties to:
/etc/spark2/conf/metrics.properties- Ensure the following properties are enabled in spark-defaults.conf for all Spark jobs:
spark.eventLog.enabled=truespark.eventLog.dir=hdfs:///spark2-history/- Apply the same properties in managed configurations for applications running on Spark 1.x and 3.x.
Result
- Spark sends metrics (including JVM metrics) to Graphite for Pulse ingestion.
Was this page helpful?