Deploy Yarn Optimizer

For the YARN Optimizer to function, the Kerberos keytab for the Pulse product must be granted the yarn rmadmin -updateNodeResource permission. For more information, see Apache Hadoop 3.3.6 - Yarn Commands.

The YARN optimizer can be safely deployed by setting the subset to include nodes available in the cluster for optimization.

The behavior of the optimization algorithm can be controlled using the following settings:

• Reserved buffer memory for non-YARN processes
• Maximum percentage of memory to be overcommitted
• Maximum step size for adjusting the amount of overcommitted memory every 5 seconds

To deploy the Yarn Optimizer server, perform the following steps:

Deploying the Add-on

1. Execute the command accelo deploy addons to start the add-on deployment process.

accelo deploy addons

2. From the list of available components, choose the Yarn Optimizer service by navigating with the arrow keys and selecting it.

[root@sac04:~ (ad-default)]$ accelo deploy addons

WARN: Gauntlet is running in dry run mode. Disable this to delete indices from elastic and purge data from mongo DB

INFO: Active Cluster:  zeus

? Select the components you would like to install:   [Use arrows to move, space to select, <right> to all, <left> to none, type to filter]

  [ ]  Oozie Connector

  [ ]  Proxy

  [ ]  QUERY ROUTER DB

  [ ]  Recommendation Service

  [ ]  SHARD SERVER DB

  [ ]  StandAlone Connector

> [X]  Yarn optimizer

3. Press Enter to deploy the Yarn Optimizer container.

Generating the Docker Configuration File:

1. Generate the Docker configuration file by running accelo admin makeconfig ad-yarn-optimizer. This action creates a necessary configuration file for the Yarn Optimizer.

accelo admin makeconfig ad-yarn-optimizer

2. Once generated, you'll need to review and possibly edit the file located at /data01/acceldata/config/docker/addons/ad-yarn-optimizer.yml. This step ensures the settings align with your specific requirements.

Output:

[root@sac04:~ (ad-default)]$ accelo admin makeconfig ad-yarn-optimizer

WARN: Gauntlet is running in dry run mode. Disable this to delete indices from elastic and purge data from mongo DB

✓ Done, Configuration file generated

IMPORTANT: Please edit/verify the file '/data01/acceldata/config/docker/addons/ad-yarn-optimizer.yml'.

If the addon is already up and running, use './accelo deploy addons' to remove and recreate the addon service.

[root@sac04:~ (ad-default)]$ cat /data01/acceldata/config/docker/addons/ad-yarn-optimizer.yml

version: "1"

services:

  ad-yarn-optimizer:

    image: ad-yarn-optimizer

    container_name: ""

    environment:

    - MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXqyScAm+LrS8Y9EWT8A8/30

    - NATS_HOST=ad-events

    volumes:

    - /etc/localtime:/etc/localtime:ro

    - /data01/acceldata/config/krb/security:/krb/security

    - /etc/hosts:/etc/hosts:ro

    ulimits: {}

    ports:

    - 19888:9888

    depends_on: []

    opts: {}

    restart: ""

    extra_hosts: []

    network_alias: []

label: Yarn optimizer

Restarting the Docker Container:

1. To restart the Yarn Optimizer container at any point, use the command accelo restart ad-yarn-optimizer.

accelo restart ad-yarn-optimizer

Deploy the Yarn Metrics Agent via Ansible

Deploying the Yarn Metrics Agent:

1. Ensure any older versions of the agents are uninstalled by executing accelo uninstall remote.

accelo uninstall remote

2. Deploy the new agents using accelo deploy addons.

accelo deploy addons

Configuring the Yarn Metrics Agent:

Using the HYDRA HOSTS YAML File:

To modify the yarn_opt_enable flag:

1. Open the hydra_hosts.yml file located at $AcceloHome/work/<clusterName>/hydra_hosts.yml.
2. Change the yarn_opt_enable value under the vars section to true.
3. Save your changes.
4. Uninstall the agents that are already running using accelo uninstall remote.
5. Deploy the agents again with the new config: accelo deploy hydra.

Using the VARS YAML File:

To enable the Yarn Metrics Agent when the yarn_opt_enable flag is set to false:

1. Create an override.yml file if it doesn't exist:

touch $AcceloHome/work/<clustername>/override.yml

2. Open the override.yml file and add the following configuration, ensuring the yarn_opt_enable is set to true.

vi $AcceloHome/work/<clustername>/override.yml

base:

    yarn_opt_enable: true

3. Save the file and run accelo reconfig cluster to apply the new configuration.

accelo reconfig cluster

Verifying the Agent's Status:

To confirm if the agent is operational, perform the following steps:

1. Log into one of the YARN Node Manager nodes and run systemctl status pulseyarnmetrics.

systemctl status pulseyarnmetrics

2. For log details, use journalctl -u pulseyarnmetrics.

journalctl -u pulseyarnmetrics

Sample Output:

[root@hdp201 ~]# journalctl -u pulseyarnmetrics

-- Logs begin at Mon 2024-01-15 17:22:26 IST, end at Fri 2024-03-22 15:47:45 IST. --

Mar 21 22:10:23 hdp201.acceldata.dvl systemd[1]: Started YARN metrics collector service.

Mar 21 22:10:23 hdp201.acceldata.dvl yarnmetrics[37475]: Cannot load the yarnMetric config: '/opt/pulse/yarnmetrics/config/yarnmetrics.conf'. Because: unexpecte

Mar 21 22:10:25 hdp201.acceldata.dvl systemd[1]: pulseyarnmetrics.service holdoff time over, scheduling restart.

Mar 21 22:10:25 hdp201.acceldata.dvl systemd[1]: Stopped YARN metrics collector service.

Mar 21 22:10:25 hdp201.acceldata.dvl systemd[1]: Started YARN metrics collector service.

Mar 21 22:10:25 hdp201.acceldata.dvl yarnmetrics[37594]: Cannot load the yarnMetric config: '/opt/pulse/yarnmetrics/config/yarnmetrics.conf'. Because: unexpecte

Mar 21 22:10:26 hdp201.acceldata.dvl systemd[1]: pulseyarnmetrics.service holdoff time over, scheduling restart.

Mar 21 22:10:26 hdp201.acceldata.dvl systemd[1]: Stopped YARN metrics collector service.

Mar 21 22:10:26 hdp201.acceldata.dvl systemd[1]: Started YARN metrics collector service.

Mar 21 22:10:26 hdp201.acceldata.dvl yarnmetrics[37656]: Cannot load the yarnMetric config: '/opt/pulse/yarnmetrics/config/yarnmetrics.conf'. Because: unexpecte

Mar 21 22:10:27 hdp201.acceldata.dvl systemd[1]: pulseyarnmetrics.service holdoff time over, scheduling restart.

Mar 21 22:10:27 hdp201.acceldata.dvl systemd[1]: Stopped YARN metrics collector service.

Mar 21 22:10:27 hdp201.acceldata.dvl systemd[1]: Started YARN metrics collector service.

Mar 21 22:10:27 hdp201.acceldata.dvl yarnmetrics[37759]: Cannot load the yarnMetric config: '/opt/pulse/yarnmetrics/config/yarnmetrics.conf'. Because: unexpecte

Mar 21 22:10:29 hdp201.acceldata.dvl systemd[1]: pulseyarnmetrics.service holdoff time over, scheduling restart.

Mar 21 22:10:29 hdp201.acceldata.dvl systemd[1]: Stopped YARN metrics collector service.

Mar 21 22:10:29 hdp201.acceldata.dvl systemd[1]: Started YARN metrics collector service.

Mar 21 22:10:29 hdp201.acceldata.dvl yarnmetrics[37942]: Cannot load the yarnMetric config: '/opt/pulse/yarnmetrics/config/yarnmetrics.conf'. Because: unexpecte

Mar 21 22:10:30 hdp201.acceldata.dvl systemd[1]: pulseyarnmetrics.service holdoff time over, scheduling restart.

Deploy Agents via HYStaller

If you are installing agents through HYStaller, follow these instructions:

Environment Variable for Service Control: The YARN_OPT_ENABLE variable is used to enable or disable the service through HYStaller.

1. Obtain HYStaller:

   1. Download the HYStaller binary from the License UI.

2. Install Hydra Agent:

   1. Before installation, replace <PULSE_HOSTNAME> with the fully qualified domain name (FQDN) of your Pulse server.
   2. Execute the script below to stop and disable existing services before proceeding with the HYStaller installation:

((sudo systemctl stop hydra && sudo systemctl disable hydra || true) && (sudo systemctl stop pulsenode && sudo systemctl disable pulsenode || true) && (sudo systemctl stop pulsejmx && sudo systemctl disable pulsejmx || true) && (sudo systemctl stop pulselogs && sudo systemctl disable pulselogs || true) || true)

sudo chown root:root /tmp/hystaller

sudo chmod 0700 /tmp/hystaller

ls -l /tmp/hystaller

sudo /tmp/hystaller uninstall

3. Configure and Install: Set up your environment variables and execute the HYStaller with the install command:

PULSE_HOME="/opt/pulse"

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"

HYDRA_SERVER_URL="http://<PULSE_HOSTNAME>:19072"

HYDRA_HEARTBEAT_DURATION="60"

HYDRA_PARCEL_MODE="False"

HYDRA_HOSTNAME_CASE="lower"

HYDRA_HOSTNAME_METHOD="CMD"

HYDRA_HEARTBEAT_JITTER="10"

YARN_OPT_ENABLE="false"

​

sudo env "PULSE_HOME=$PULSE_HOME" "PATH=$PATH" "HYDRA_SERVER_URL=$HYDRA_SERVER_URL" "HYDRA_HEARTBEAT_DURATION=$HYDRA_HEARTBEAT_DURATION" "HYDRA_PARCEL_MODE=$HYDRA_PARCEL_MODE" "HYDRA_HOSTNAME_CASE=$HYDRA_HOSTNAME_CASE" "HYDRA_HOSTNAME_METHOD=$HYDRA_HOSTNAME_METHOD" "HYDRA_HEARTBEAT_JITTER=$HYDRA_HEARTBEAT_JITTER" /tmp/hystaller install

Configure User Account for Running the Pulseyarnmetrics Agent

Perform the following steps:

1. Update the User Configuration in vars.yml:
   1. Locate the agent parameter within the vars.yml file and change the yarnmetrics_user value to the desired username (default is "yarn"):

agent:

    yarnmetrics_user: newuser

2. Create or Update the override.yml File:

• If absent, generate a new override.yml file within your cluster's working directory:

touch $AcceloHome/work/<clustername>/override.yml

• Edit the override.yml file to include the updated user configuration:

vi $AcceloHome/work/<clustername>/override.yml

• Ensure the file contains the following lines, adjusting the yarnmetrics_user as necessary:

agent:

    yarnmetrics_user: newuser

3. Apply Configuration Changes: Save the modifications and execute the accelo reconfig cluster command to update the cluster with the new user settings.

accelo reconfig cluster

4. Verify the Change:

• Check the systemd unit file on one of the nodes to confirm the new user is specified:

cat /etc/systemd/system/pulseyarnmetrics.service

• The User field in the service configuration should reflect the new user account chosen.

Configure the OverCommit Timeout Value and Node ID Port

Follow the steps to configure the overcommit timeout value to revert the changes if NATs are unreachable. Also, you can configure the Node ID port. This configuration helps you to revert each node to its original value if nats is unreachable for a specific amount of time.

To fetch the node ID port run the command yarn node -list in any cluster node.

The parameters in vars.yml file are as follows. By default, the OverCommit timeout value is set to 300 seconds.

base:

    yarn_opt_overcommit_timeout_enabled: "true"

    yarn_opt_overcommit_timeout_seconds: "300"

    yarn_nodeid_port: "45454"

1. Create the override.yml file if not present already.

touch $AcceloHome/work/<clustername>/override.yml

2. Open the override.yml file.

vi $AcceloHome/work/<clustername>/override.yml

3. Put the following configuration in the override.yml file if not present already.

base:

  yarn_opt_overcommit_timeout_enabled: "true"

  yarn_opt_overcommit_timeout_seconds: "100"

  yarn_nodeid_port: "8041"

4. Save the file.
5. Run the reconfig cluster.

accelo reconfig cluster

6. Verify the config file for the pulseyarnmetrics agent.

cat /opt/pulse/yarnmetrics/config/yarnmetrics.conf

{

  "clusterName": "odp_zoro",

  "resourcemanagerIP": "odp102.acceldata.dvl",

  "resourcemanagerPort": 8088,

  "isKerberosEnabled": false,

  "keytabPath": "/opt/pulse/node/config/node.keytab",

  "kerberosPrinciple": "hdfs@ACCELDATA.COM",

  "kerberosDateFormat": "01/02/2006",

  "victoriaDBEndpoint": "http://plat02.acceldata.dvl:19043/insert/1385609323/influx",

  "natsEndpoint": "http://plat02.acceldata.dvl:19009",

  "natsConsumerAckWait": 20,

  "isMetricsProcessingNode": false

  "overcommit_timeout_enabled": true,

  "overcommit_timeout_seconds": 100,

  "nodeIdPort": 8041

}

Enable REST API call for Overcommitment of Node Memory

To enable the Rest API call, perform the following steps:

1. Run the following command.

accelo admin makeconfig ad-yarn-optimizer

2. Add the following environment variable in the ad-yarn-optimizer.yaml file.

IS_REST_EVENT_SUPPORTED=true

Enable Node Limit as Configured in YARN Configurations

You can follow the below steps to consider the memory limit configured in the Ambari UI > Yarn Configurations.

Follow the below steps to enable this feature:

1. Set the environment variable to true in ad-yarn-optimizer.yml.

ENABLE_LIMIT YARN_STATIC_CONFIG to true

2. Restart the service using the following command.

accelo restart ad-yarn-optimizer

Enable Static Queue Optimization

Follow the steps to enable the static Queue Optimization.

1. SET "ENABLE_STATIC_QUEUES_OVERCOMMITMENT" environment variable to True in ad-yarn-optimizer.yml.
2. In case of the CDP setup, set "QUEUE_UPDATE_API" to "/ws/v1/cluster/scheduler-conf?user.name=hdfs" in ad-yarn-optimizer.yml.
3. Restart the ad-yarn-optimizer service by running the following command.

accelo restart ad-yarn-optimizer

4. Set enableStaticQueue to True in the FEATURE_FLAGS environment variable in ad-graphql.