Pulse Multi-Node Rancher Based Deployment

This document explains the process of deploying Pulse on a single node Kubernetes (K8s). This deployment is performed by using Rancher.

Prerequisites

• All the servers must be part of the same network.

• You must manually set the hostname of the servers you create by executing the command hostnamectl set-hostname pulk8s01.sre.axl

• Do ssh copy-id in all the list of IPs which is provided from datacenter.

  • If you do not have a key pair in your system, you must execute the command ssh-keygen.
  • ssh-copy-id root@<<IP address>>

• Install Kubectl from the official Kubernetes documentation.

Rancher Cluster Setup in K8s

To set up a rancher based cluster:

1. Clone the GIT repository git@bitbucket.org:acceldata/ad-platform-infrastructure.git.
2. Copy the public key to the internal_infra folder. You can navigate to this folder by executing the following command. You must use the same key pair that was generated above.

xxxxxxxxxx

cd ad-platform-infrastructure/ansible/internal_infra

3. Edit the docker.yml, content. You must replace the id_rsa.pub, with the name of the file as shown in the following block

xxxxxxxxxx

# Change

authorized_key:

        user: docker

        state: present

        key: "{{ lookup('file', 'id_rsa.pub') }}"

# To

authorized_key:

        user: docker

        state: present

        key: "{{ lookup('file', 'popk8s.pub') }}"

4. Edit the docker password as shown in the following block.

xxxxxxxxxx

# Change

name: Add a new user named docker

      user:

        name: docker

        groups: docker

        password:

# To 

name: Add a new user named docker

      user:

        name: docker

        groups: docker

        password: 24242sfsfad##$

5. Save and exit the docker.yml file.
6. Edit the hosts.ini file with the fully qualified domain name (FQDN) and the Internet Protocol (IP) and update the key. You can accomplish this by executing the following block.

xxxxxxxxxx

[popk8s]

popk8smultinode1000.qe.axl.iti  ansible_host=<<ip address>>

popk8smultinode1001.qe.axl.iti  ansible_host=<<ip address>>

popk8smultinode1002.qe.axl.iti  ansible_host=<<ip address>>

[all:vars]

ansible_user=root

ansible_python_interpreter=/usr/bin/python

ansible_ssh_private_key_file=~/.ssh/popk8s

7. Test if the ping command works, based on the group name provided. You can accomplish this by executing the following command.

xxxxxxxxxx

ansible -m ping popk8s

8. Execute the following command to install docker and ntp in all the machines that are part of the group popk8s. You must execute the command in all the machines that are part of the popk8s group.

xxxxxxxxxx

ansible-playbook docker.yml --extra-vars="nodes=popk8s"

Install Rancher

1. Download and install Ranched by executing the following block of code.

xxxxxxxxxx

wget https://github.com/rancher/rke/releases/download/v1.3.3/rke_linux-amd64

chmod +x rke_linux-amd64

mv rke_linux-amd64 /usr/local/bin/rke

rke --version

>>rke version v1.3.3

2. Configure RKE/Rancher to bring up Kubernetes cluster. You can accomplish this by executing the following block.

# Create a directory by executing the following command.

mkdir popk8smultinodecluster

# Change the directory by executing the following command. 

bash-3.2$ cd popk8smultinodecluster/

#enter password

bash-3.2$ pwd

/<<filepath>>/popk8smultinodecluster

#Configuratrion command 

bash-3.2$ rke config

# You are asked the following questions during installation. You can view the sample responses given by Acceldata. You can modify the responses as per your organization requirements. 

[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/popk8s

[+] Number of Hosts [1]: 3

[+] SSH Address of host (1) [none]: <<ip address>>

[+] SSH Port of host (1) [22]:

[+] SSH Private Key of host (<<ip address>>) [none]: ~/.ssh/popk8s

[+] SSH User of host (<<ip address>>) [ubuntu]: docker

[+] Is host (<<ip address>>) a Control Plane host (y/n)? [y]: y

[+] Is host (<<ip address>>) a Worker host (y/n)? [n]: y

[+] Is host (<<ip address>>) an etcd host (y/n)? [n]: y

[+] Override Hostname of host (<<ip address>>) [none]:

[+] Internal IP of host (<<ip address>>) [none]:

[+] Docker socket path on host (<<ip address>>) [/var/run/docker.sock]:

[+] SSH Address of host (2) [none]: <<ip address>>

[+] SSH Port of host (2) [22]:

[+] SSH Private Key Path of host (<<ip address>>) [none]: ~/.ssh/popk8s

[+] SSH User of host (<<ip address>>) [ubuntu]: docker

[+] Is host (<<ip address>>) a Control Plane host (y/n)? [y]:

[+] Is host (<<ip address>>) a Worker host (y/n)? [n]: y

[+] Is host (<<ip address>>) an etcd host (y/n)? [n]: y

[+] Override Hostname of host (<<ip address>>) [none]:

[+] Internal IP of host (<<ip address>>) [none]:

[+] Docker socket path on host (<<ip address>>) [/var/run/docker.sock]:

[+] SSH Address of host (3) [none]: <<ip address>>

[+] SSH Port of host (3) [22]:

[+] SSH Private Key Path of host (<<ip address>>) [none]: ~/.ssh/popk8s

[+] SSH User of host (<<ip address>>) [ubuntu]: docker

[+] Is host (<<ip address>>) a Control Plane host (y/n)? [y]: y

[+] Is host (<<ip address>>) a Worker host (y/n)? [n]: y

[+] Is host (<<ip address>>) an etcd host (y/n)? [n]: y

[+] Override Hostname of host (<<ip address>>) [none]:

[+] Internal IP of host (<<ip address>>) [none]:

[+] Docker socket path on host (<<ip address>>) [/var/run/docker.sock]:

[+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: weave

[+] Authentication Strategy [x509]:

[+] Authorization Mode (rbac, none) [rbac]:

[+] Kubernetes Docker image [rancher/hyperkube:v1.21.7-rancher1]:

[+] Cluster domain [cluster.local]:

[+] Service Cluster IP Range [<<ip address>>/16]:

[+] Enable PodSecurityPolicy [n]:

[+] Cluster Network CIDR [<<ip address>>/16]:

[+] Cluster DNS Service IP [<<ip address>>]:

[+] Add addon manifest URLs or YAML files [no]:

bash-3.2$ ls

cluster.yml popk8s

3. Edit the cluster.yml file. You must replace the entire ingress block the following block.

xxxxxxxxxx

ingress:

  provider: none

4. Create the K8s cluster by executing the following code.

xxxxxxxxxx

bash-3.2$ rke up

You get the following message on screen.

xxxxxxxxxx

INFO[0000] Running RKE version: v1.3.3                  

INFO[0000] Initiating Kubernetes cluster                

INFO[0000] [dialer] Setup tunnel for host [<<ip address>>] 

INFO[0000] [dialer] Setup tunnel for host [<<ip address>>] 

INFO[0000] [dialer] Setup tunnel for host [<<ip address>>] 

INFO[0001] Checking if container [cluster-state-deployer] is running on host [<<ip address>>], try #1

INFO[0001] Pulling image [rancher/rke-tools:v0.1.78] on host [<<ip address>>], try #1

Important

If you encounter any error in getting the K8s cluster up, you must execute the following command in each node and reboot the node.

xxxxxxxxxx

rm -rf /etc/kubernetes/

rm -rf /var/lib/kubelet/

rm -rf /var/lib/etcd/

docker rmi $(docker images -a -q)

docker stop $(docker ps -a -q)

docker rm $(docker ps -a -q)

Important

You must now test the connectivity to the Kubernetes cluster. You may encounter an error with the message The connection to the server localhost:8080 was refused - did you specify the right host or port? . If you get this error, you must add the following command to .bashrc.

xxxxxxxxxx

export KUBECONFIG=/root/popk8smulti/kube_config_cluster.yml

5. You can check the list of hosts by executing the following command.

xxxxxxxxxx

kubectl get nodes

Note

To execute the kubectl command, you must install Kubectl tool. You can follow this documentation to install Kubectl.

Metallb and Nginx Setup

1. Download the keyz binary for ECR secret and replace the following file For MAC system.

https://bitbucket.org/acceldata/ad-automation-test/src/Dev/ops/popk8scluster/

2. Check out the Bitbucket repository https://bitbucket.org/acceldata/ad-service/src/development/.

xxxxxxxxxx

bash-3.2$ pwd

/<<filepath>>/helm-controller

bash-3.2$ bash-3.2$ ls .scripts/keyz

.scripts/keyz

bash-3.2$ chmod +x ./.scripts/keyz

##### Verification that you are using the keys relevant for MAC

bash-3.2$ file .scripts/keyz

.scripts/keyz: Mach-O 64-bit executable x86_64

3. Open the helmfile-ha.yml or helmfile.yml file.
4. Comment the lines of code as shown in the following block.

xxxxxxxxxx

hooks:

  - events: ["prepare", "presync"]

    showlogs: true

    command: ".scripts/ecr-secret-reset.sh"

    args: ["{{ .Namespace }}"]

#  - events: ["prepare"]

#    showlogs: true

#    command: ".scripts/clone-package.sh"

#    args: ["{{ .Namespace }}"]

#  - events: ["prepare"]

#    command: ".scripts/add-package.sh"

#  - events: ["cleanup"]

#    command: ".scripts/remove-package.sh"

Nginx Deployment

If you are planning multiple Pulse installations and if each installation has a different namespace, you must execute the steps in the (Link Removed) document and then proceed with the steps.

1. Execute the following commands to install the helmfile.

xxxxxxxxxx

wget (appropriate link for your system from https://github.com/roboll/helmfile/releases/tag/v0.143.3)

chmod +x helmfile_linux-amd64

mv helmfile_linux_amd64 /usr/local/bin/helmfile

helmfile --version

helmfile version v0.143.3

2. Execute the following commands to install the helm.

xxxxxxxxxx

wget https://get.helm.sh/helm-v3.9.4-linux-amd64.tar.gz

tar -xvf helm-v3.9.4-linux-amd64.tar.gz

mv linux-amd64/helm /usr/local/bin/helm

3. Execute the following commands in to install K9s.

xxxxxxxxxx

mkdir k9s-install

cd k9s-install/

curl -sS https://webinstall.dev/k9s | bash

You get the response as shown in the following block

xxxxxxxxxx

[root@pulsek8smain ~]# mkdir k9s-install 

[root@pulsek8smain ~]# cd k9s-install/

[root@pulsek8smain k9s-install]# curl -sS https://webinstall.dev/k9s | bash

Thanks for using webi to install 'k9s@stable' on 'Linux/x86_64'.

Have a problem? Experience a bug? Please let us know:

        https://github.com/webinstall/webi-installers/issues

Lovin' it? Say thanks with a Star on GitHub:

        https://github.com/webinstall/webi-installers

Downloading k9s from

https://github.com/derailed/k9s/releases/download/v0.26.3/k9s_Linux_x86_64.tar.gz

Saved as /root/Downloads/webi/k9s/0.26.3/k9s_Linux_x86_64.tar.gz

Extracting /root/Downloads/webi/k9s/0.26.3/k9s_Linux_x86_64.tar.gz

Installing to /root/.local/opt/k9s-v0.26.3/bin/k9s

Installed 'k9s v0.26.3' as /root/.local/bin/k9s

PATH.env updated with:

  export PATH="/root/.local/bin:$PATH"

TO FINISH: copy, paste & run the following command:

        source ~/.config/envman/PATH.env

        (newly opened terminal windows will update automatically)

4. Execute the following commands to complete the installation.

You can now use the helmfile by executing the following commands.

You can verify the nginx deployment by executing the following commands.

Metallb Deployment

1. Open the helm-controller values.yaml (the file location is helm-controller/ha/ad-mettlb/values.yaml)
2. Ensure you use your subnet to specify the range and not some other subnet.

3. Execute the commands in the following block to deploy Metallb.

4. Verify if metallb pods and nginix services get an External IP from the range provided.

Rook-CEPH Deployment

Execute the following commands.

Deployment of Pulse DB, Core, and Add-ons

If you are planning multiple Pulse installations and if each installation has a different namespace, you must execute the steps in the (Link Removed) document and then proceed with the steps.

1. Execute the following command for DB deployment.

2. Execute the following command for deployment of accelo and deployer.

The following sample commands must be used to push license file, keytab, certs, krb5.conf to pvc.

For HDP or Kafka standalone (ansible deployment), you must generate an SSH key and ensure that you are able to perform passwordless secure login to the cluster nodes, and then copy the key to the ad-accelo pod. sample command. The sample command is as follows.

3. Execute inside the ad-accelo-configure pod and verify below ENV’s.

4. Configure core in the accelo pod by executing the following command.

5. In the pod execute the following command.

You get the message as shown in the following block.

The deployer asks the following questions. You can respond to them according to your configurations.

The following block shows sample responses for each of the above questions.

The sample discovered configurations window is as follows. You can also find the next set of questions asked.

Notes

• The IP addresses for Hydra is same as the container name, To Fix this issue, create a override.yml file in the /data01/acceldata/work/<Cluster_Name>/ location.
• The cluster name for Victoria metrics is obtained from the acceldata__advisa__test.conf. You must replace it in override.yml file.
• You must add the following environment.

The content of override.yml file is as follows.

Mongo DB URI Correction

1. Execute the following commands in accelo pod.

2. Replace the MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXosC8vfVkGYGWGPNnX64ZVSp9yHgErQknPBAfYZ9cOG1A== in the /data01/acceldata/config/acceldata<clusterName>.conf file.

For ansible based deployment of hydra (HDP/kafka), fix hydra_server_ ur manually by executing the following code block.

3. Execute the following command.

Deploy Core Services

1. Uncomment the following statements from helmfile-ha/helmfile.yml file, You must perform this step for all core and addon services. You must retain the comment if you wish to deploy rook-ceph, ad-accelo.

2. The version/build number currently used is containers.tagName=2.1.9-200622. You must execute the following command.

Deploy Add-on Services

1. Execute the following command to deploy the add on services.

You can now access the Pulse UI from the following URL.

FSA Load

You must execute these commands after configuring the add-on services.

1. Connect to ad-accelo-configure, pod and run the following command.

2. Execute the folloiwng to configure notifications.

3. Validate if the ad-hydra server container is running and then execute the following command.

Other Configurations

Installation and Uninstallation of agents for HDP/Kafka (ansible based deployment). Once sync is run you must delete it before running with “uninstall tag“.

1. Ensure that the /ad-service/ops/helm-controller/helmfile-ha.yaml file has the following content.

Note : clusterName is the same name that you provided in the accelo config cluster.

2. Execute the commands in the following block to monitor weave scope.

You can access the weave UI from the http://localhost:4040 UI.

3. Execute the following command to delete the add-ons.

4. To deploy a new version of the build, execute the following command (Consider that the current version is 2.1.9-200622 and you want to upgrade to 2.1.9-200822).

Configurations for Kafka Standalone

For Kafka you need not deploy all the add-on services. You must add a new selection kafkastandalone: true.

2. Execute the following commands to deploy addons for Kafka.

3. The override.yaml file must have the following content. This file requires a access reconfigured cluster after the changes are done. This required for logs agent to understand and read the log files for Kafka.