New Heartbeat for Pulse

Acceldata is delighted to announce the introduction of Hydra as a perfect installation technique for Pulse. With the introduction of Hydra, Pulse is now enhanced with a much simpler method for installing and maintaining Pulse Agents in Hadoop clusters. Hydra brings in the greatest change by allowing user based deployment and maintenance. This advancement propels Pulse to the next advancement levels.

A handful of our users were unable to grant SSH access to deploy Pulse in their environments due to security and regulatory concerns. Now with the arrival of Hydra, the need for SSH access to deploy Pulse will no longer be a point of concern. As an outcome, Acceldata achieves one of the most significant advances in Pulse version 2.0.

Hydra Agent is available for both CDP and non-CDP platforms, and users can easily deploy and maintain it. For CDP clusters, Hydra Agent is available as a parcel. Cloudera Manager is used to deploy and configure it. The Hydra Server will act as both a configuration management server and a parcel repository. For non-CDP clusters, Hydra Agent is available as an RPM/DEB package that users can distribute and install on their cluster nodes using their preferred configuration management tool.

Prior to Hydra

SSH access was required in two situations:

1. Installing the Pulse agents and dependencies - a one-time task.
2. Each agent's configuration files must be updated - executed from time to time.

The problem at hand is that due to security and compliance concerns, SSH access is not permitted in some user environments. The Pulse Agents are difficult to deploy in these environments. Even if we obtain one-time SSH access for Agent installation, we will be unable to update the agent configurations on a regular basis as the cluster configuration or state changes.

Hydra is developed as a solution for the deployment and maintenance issues in non-SSH environments. The Hydra component is made up of two services: a server and an agent.

In the Pulse server node, the Hydra Server will be installed as an add-on. This will serve as the Hydra Agent CDP parcel's configuration management server and repository provider.

For CDP Clusters

For deploying Hydra Agent to a CDP cluster, the Hydra Agent should be distributed as a Cloudera Parcel and run as a CDP service (managed by Cloudera Manager) in all Hadoop cluster nodes. For the Hydra Agent Parcel, the Hydra Server service running on the Pulse Server node will act as a parcel repository. The user will be able to download, distribute, and install the Hydra Agent Parcel after adding the Hydra Server URL as a repository in Cloudera Manager. This also enables Cloudera Manager to upgrade the Hydra Agent.

For Non-CDP Clusters

The SSH access requirement is not completely removed for non-CDP cluster deployments, but it is still required for non-CDP-based deployments. Installing Pulse Agent on Hadoop clusters for non-CDP clusters can be categorized into two different ways:

• If the user is prepared to offer one-time SSH access and password-less sudo privilege, we can deploy the Hydra Agent to the cluster nodes using the Pulse CLI (accelo). Once installed, the Hydra Agent will keep the configuration files up-to-date and restart the agent services as needed.
• If the user is not willing to provide SSH Access, then the Hydra Agent will be delivered as an RPM/DEB package, and users can distribute and install it in their Hadoop cluster nodes using whatever configuration management tool they prefer. Users need to set certain environmental variables while installing the agent in order for the Hydra Agent to discover the Hydra Server in the Pulse Server node.

When we have a new feature/patch for the Hydra Agent, we may need to upgrade the Hydra Agent. In these scenarios, SSH access will be required to upgrade the Hydra Agent.

With Hydra as our solution, SSH access is no longer required for CDP-based deployments, but it is partially removed for non-CDP-based deployments. One can obtain the current status of each agent's service. Whether the agent service is running or not, Hydra Agent sends the heartbeat. New agent versions and agent-specific features are automatically deployed (except the Hydra Agent itself). It is one step further in automating the reconfiguration of Pulse when the cluster state changes.