Fetch K8s Metrics in an OpenShift Cluster

For Spark on K8S observability, Pulse provides charts and dashboards for Kubernetes namespace-level resource metrics such as CPU usage and memory usage.

• Kubernetes metrics can be collected using different approaches, including custom integrations or open-source monitoring solutions. In OpenShift environments, a monitoring stack is available out of the box as part of the cluster setup.
• Pulse integrates with this OpenShift monitoring stack to collect Kubernetes metrics and ingest them into its data platform, enabling visualization through charts and dashboards.
• Pulse leverages this solution and configures it to forward Kubernetes metrics to its data platform for storage and analysis. For details, see Configuring and using the monitoring stack in OpenShift Container Platform.

1. Edit the cluster-monitoring-config config map.

xxxxxxxxxx

oc edit configmaps cluster-monitoring-config -n openshift-monitoring

2. Put the following inside the config map.

xxxxxxxxxx

prometheusK8s:

      remoteWrite:

        - url: "http://<VMUsername>:<VMPassword>@<VMInsert Hostname>:19043/insert/<ClusterHash>/prometheus"

          writeRelabelConfigs:

          - action: labelmap

            regex: __meta_kubernetes_(.+)

          - sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_path]

            targetLabel: __metrics_path__

            regex: (.+)

            action: replace

          - sourceLabels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]

            targetLabel: __address__

            regex: (.+):(?:\d+);(\d+)

            replacement: ${1}:${2}

            action: replace

          - sourceLabels: [__name__]

            targetLabel: __name__

            regex: (.*)

            replacement: 'kubernetes.${1}'

            action: replace

3. Save the changes.
4. Now, when the next time prometheus-operator reconciles, it will pick up the new config. To make it faster, delete the Prometheus pods so they restart immediately with the updated configuration.

xxxxxxxxxx

oc delete pod -l prometheus=k8s -n openshift-monitoring

5. After the pods are up, you can see the metrics flowing inside the DB.