Configure ODP Flink
To enable Pulse to collect job- and application-level metrics from Flink, configure Graphite metric reporting in Ambari.
This setup allows Flink to export real-time operational data so that Pulse can monitor.
Expose Flink JMX metrics
Expose Flink metrics using a Graphite reporter, which sends metrics such as job performance, resource utilization, execution latency, etc. from Flink to Pulse.
- In the Ambari UI, go to Services > Flink > Configs.
- Under Custom flink config, click + add property to add the following configurations. If they already exist, update them as below.
metrics.reporter.graphite.factory.class= org.apache.flink.metrics.graphite.GraphiteReporterFactorymetrics.reporter.graphite.host= localhostmetrics.reporter.graphite.interval=2 SECONDSmetrics.reporter.graphite.port=12005metrics.reporter.graphite.prefix=flink.metricsmetrics.reporter.graphite.protocol=TCPmetrics.reporter.graphite.protocol:TCPyarn.application.type=FLINK
- Save the changes.
Verify History Server Configuration
Before running the cluster configuration in Pulse, verify the History Server address and port in the Ambari configuration.
Navigation path: In the Ambari UI > Services > Flink > Under Configs > Advanced flink-conf.
Check the following parameters:
- historyserver.web.address
- Historyserver.web.port
The ad-flink-connector uses these configurations to poll job-related data from the Flink History Server.
After verifying these settings, run the following command in the Pulse Accelo CLI to proceed with the ad-flink connector deployment.
accelo config clusterOnce configured, push the changes by running the command:
accelo admin database push-config -aIf you change the History Server URL or port in Ambari, you must update the configuration to ensure that the new values are reflected in the ad-flink-connector.
To do this, run the following commands in sequence on the Pulse node:
- accelo config cluster > to configure the details.
- accelo admin database push-config -a > to push the changes.
Do not use the accelo reconfig cluster command for this update.
Running accelo reconfig cluster does not refresh the connector configuration with the updated History Server details from Ambari.
Always run accelo config cluster followed by accelo admin database push-config -a to ensure that the ad-flink-connector configuration is correctly updated with the latest History Server information.
Grant User Access in Ranger (If Access Error Occurs)
If you encounter a Kerberos or authentication exception indicating that a user cannot access the Flink directory or submit a Flink job, update the Ranger YARN policy to grant the required access.
Steps:
- In the Ranger UI, go to Service Manager > YARN > odp_*_yarn Policies.
- Edit the policy for your environment (for example, odp_kafka3x_yarn).
- Under Allow Conditions, add the user who is running the Flink job in the Select Users column.
- Assign the necessary Permissions, such as submit-app and admin-queue.
- Click Save to apply the policy changes.
This ensures the user has the appropriate access to YARN resources and directories for Flink and Pulse integration.