Pulse Single Node Rancher Based Deployment

This document explains the process of deploying Pulse on a single node Kubernetes (K8s). This deployment is performed by using Rancher.

Prerequisites

• All the servers must be part of the same network.

• You must manually set the hostname of the servers you create.

• You must peform ssh copy-id in all the list of IPs which is provided from datacenter.

  • If you do not have a key pair in your system, you must execute the command ssh-keygen.
  • ssh-copy-id root@<IP address>>

• Install Kubectl from the official Kubernetes documentation.

Rancher Cluster Setup in K8s

To set up a rancher based cluster:

1. Clone the GIT repository git@bitbucket.org:acceldata/ad-platform-infrastructure.git.
2. Copy the public key to the internal_infra folder. You can navigate to this folder by executing the following command. You must use the same key pair that was generated above.

xxxxxxxxxx

cd ad-platform-infrastructure/ansible/internal_infra

3. Edit the docker.yml content. You must replace the id_rsa.pub with the name of the file as shown in the following code block:

xxxxxxxxxx

# Change

authorized_key:

        user: docker

        state: present

        key: "{{ lookup('file', 'id_rsa.pub') }}"

# To

authorized_key:

        user: docker

        state: present

        key: "{{ lookup('file', 'popk8s.pub') }}"

4. Edit the docker password as shown in the following block.

xxxxxxxxxx

# Change

name: Add a new user named docker

      user:

        name: docker

        groups: docker

        password:

# To 

name: Add a new user named docker

      user:

        name: docker

        groups: docker

        password: 

##$

5. Save and exit the docker.yml file.
6. Edit the hosts.ini file with the fully qualified domain name (FQDN), Internet Protocol (IP), and update the key. You can accomplish this by executing the following:

xxxxxxxxxx

[popk8s]

<<hostname>>  ansible_host=<<ip address>

[all:vars]

ansible_user=root

ansible_python_interpreter=/usr/bin/python

ansible_ssh_private_key_file=~/.ssh/popk8s

7. Test if the ping command works, based on the group name provided by running the following command:

xxxxxxxxxx

ansible -m ping popk8s

8. Execute the following command to install Docker and ntp in all the machines that are part of the group popk8s. You must execute the command in all the machines that are part of the popk8s group.

xxxxxxxxxx

ansible-playbook docker.yml --extra-vars="nodes=popk8s"

Install Rancher

1. Download and install Ranched by executing the following block of code.

xxxxxxxxxx

wget https://github.com/rancher/rke/releases/download/v1.3.3/rke_linux-amd64

chmod +x rke_linux-amd64

mv rke_linux-amd64 /usr/local/bin/rke

rke --version

>>rke version v1.3.3

2. Configure RKE/Rancher to bring up the Kubernetes cluster by running the following code:

# Create a directory by executing the following command.

mkdir popk8smultinodecluster

# Change the directory by executing the following command. 

bash-3.2$ cd popk8smultinodecluster/

#enter password

bash-3.2$ pwd

/<<filepath>>/popk8smultinodecluster

# Configuratrion command 

bash-3.2$ rke config

# You are asked the following questions during installation. You can view the sample responses given by Acceldata. You can modify the responses as per your organization requirements. 

[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/popk8s

[+] Number of Hosts [1]: 1

[+] SSH Address of host (1) [none]: <<ip address>>

[+] SSH Port of host (1) [22]:

[+] SSH Private Key of host <<ip address>> [none]: ~/.ssh/popk8s

[+] SSH User of host <<ip address>> [ubuntu]: docker

[+] Is host <<ip address>> a Control Plane host (y/n)? [y]: y

[+] Is host <<ip address>> a Worker host (y/n)? [n]: y

[+] Is host <<ip address>> an etcd host (y/n)? [n]: y

[+] Override Hostname of host <<ip address>> [none]:

[+] Internal IP of host <<ip address>> [none]:

[+] Docker socket path on host <<ip address>> [/var/run/docker.sock]:

[+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: weave

[+] Authentication Strategy [x509]:

[+] Authorization Mode (rbac, none) [rbac]:

[+] Kubernetes Docker image [rancher/hyperkube:v1.21.7-rancher1]:

[+] Cluster domain [cluster.local]:

[+] Service Cluster IP Range [<<ip address>>/16]:

[+] Enable PodSecurityPolicy [n]:

[+] Cluster Network CIDR [<<ip address>>/16]:

[+] Cluster DNS Service IP [<<ip address>>]:

[+] Add addon manifest URLs or YAML files [no]:

bash-3.2$ ls

cluster.yml popk8s

3. Edit the cluster.yml file. You must replace the entire ingress block with the following:

xxxxxxxxxx

ingress:

  provider: none

4. Create the K8s cluster by executing the following:

xxxxxxxxxx

bash-3.2$ rke up

The following message is displayed:

xxxxxxxxxx

INFO[0000] Running RKE version: v1.3.3                  

INFO[0000] Initiating Kubernetes cluster                

INFO[0000] [dialer] Setup tunnel for host [<<ip address>>] 

INFO[0001] Checking if container [cluster-state-deployer] is running on host [<<ip address>>], try #1

INFO[0001] Pulling image [rancher/rke-tools:v0.1.78] on host [<<ip address>>], try #1

Important If you encounter any error while bringing up the K8s cluster, you need to execute the following command on each node and then reboot the node.

xxxxxxxxxx

rm -rf /etc/kubernetes/

rm -rf /var/lib/kubelet/

rm -rf /var/lib/etcd/

docker rmi $(docker images -a -q)

docker stop $(docker ps -a -q)

docker rm $(docker ps -a -q)

Important Now, test the connectivity to the Kubernetes cluster. If you encounter an error message such as "The connection to the server localhost:8080 was refused - did you specify the right host or port?", you must add the following command to your .bashrc file.

xxxxxxxxxx

export KUBECONFIG=/root/popk8smulti/kube_config_cluster.yml

5. Check the list of hosts by executing the following command:

xxxxxxxxxx

kubectl get nodes

Note To use the kubectl command, you need to install the Kubectl tool. Refer to the documentation for instructions on installing Kubectl.

Metallb and Nginx Setup

1. Download the keyz binary for ECR secret and replace the following file for Mac system.

https://bitbucket.org/acceldata/ad-automation-test/src/Dev/ops/popk8scluster/

2. Check out the Bitbucket repository: https://bitbucket.org/acceldata/ad-service/src/development/.

xxxxxxxxxx

bash-3.2$ pwd

/<<filepath>>/ad-service/ops/helm-controller

bash-3.2$ bash-3.2$ ls .scripts/keyz

.scripts/keyz

bash-3.2$ chmod +x ./.scripts/keyz

##### Verification that you are using the keys relevant for MAC

bash-3.2$ file .scripts/keyz

.scripts/keyz: Mach-O 64-bit executable x86_64

3. Open the helmfile.yml file.
4. Comment the lines of code as shown in the following code block:

xxxxxxxxxx

hooks:

  - events: ["prepare", "presync"]

    showlogs: true

    command: ".scripts/ecr-secret-reset.sh"

    args: ["{{ .Namespace }}"]

#  - events: ["prepare"]

#    showlogs: true

#    command: ".scripts/clone-package.sh"

#    args: ["{{ .Namespace }}"]

#  - events: ["prepare"]

#    command: ".scripts/add-package.sh"

#  - events: ["cleanup"]

#    command: ".scripts/remove-package.sh"

Nginx Deployment

If you intend to install Pulse in multiple namespaces, follow the instructions in the (Link Removed) document before proceeding with the next steps.

1. Execute the following commands to install the helmfile.

xxxxxxxxxx

wget (appropriate link for your system from https://github.com/roboll/helmfile/releases/tag/v0.143.3)

chmod +x helmfile_linux-amd64

mv helmfile_linux_amd64 /usr/local/bin/helmfile

helmfile --version

helmfile version v0.143.3

2. Execute the following commands to install helm.

xxxxxxxxxx

wget https://get.helm.sh/helm-v3.9.4-linux-amd64.tar.gz

tar -xvf helm-v3.9.4-linux-amd64.tar.gz

mv linux-amd64/helm /usr/local/bin/helm

3. Execute the following commands to install K9s.

xxxxxxxxxx

mkdir k9s-install

cd k9s-install/

curl -sS https://webinstall.dev/k9s | bash

You get a response as shown below:

xxxxxxxxxx

mkdir k9s-install 

cd k9s-install/

curl -sS https://webinstall.dev/k9s | bash

Thanks for using webi to install 'k9s@stable' on 'Linux/x86_64'.

Have a problem? Experience a bug? Please let us know:

        https://github.com/webinstall/webi-installers/issues

Lovin' it? Say thanks with a Star on GitHub:

        https://github.com/webinstall/webi-installers

Downloading k9s from

https://github.com/derailed/k9s/releases/download/v0.26.3/k9s_Linux_x86_64.tar.gz

Saved as /root/Downloads/webi/k9s/0.26.3/k9s_Linux_x86_64.tar.gz

Extracting /root/Downloads/webi/k9s/0.26.3/k9s_Linux_x86_64.tar.gz

Installing to /root/.local/opt/k9s-v0.26.3/bin/k9s

Installed 'k9s v0.26.3' as /root/.local/bin/k9s

PATH.env updated with:

  export PATH="/root/.local/bin:$PATH"

TO FINISH: copy, paste & run the following command:

        source ~/.config/envman/PATH.env

        (newly opened terminal windows will update automatically)

4. Execute the following commands to complete the installation.

xxxxxxxxxx

source ~/.config/envman/PATH.env

k9s

You can now use the helm file by executing the following commands:

helmfile --interactive --file helmfile.yaml -l type=network  -e on-prem/dev   --state-values-set 'host=remote','containers.tagName=<<release tag>>'   -n <namespace-of-new-pulse>  sync

helmfile --interactive --file helmfile.yaml -e on-prem/dev -l app=ad-nginx-controller --state-values-set 'containers.tagName=<<release tag>>','host=remote' -n <namespace-of-new-pulse> sync

Verify the nginx deployment by executing the following commands:

xxxxxxxxxx

kubectl get po -n <name-of-new namespace> -o wide

kubectl get svc -n <name-of-new namespace> -o wide

## You must be able to view a list of nginx related pods and services.

Metallb Deployment

1. Open the helm-controller values.yaml from the file location at helm-controller/ha/ad-mettlb/values.yaml
2. Ensure to use your subnet to specify the range and not any other subnet.

3. Execute the commands in the following block to deploy Metallb.

4. Verify if metallb pods and nginix services get an External IP from the range provided.

5. For local storage deployment, execute the following command.

Deployment of Pulse DB, Core, and Add-ons

If you intend to install Pulse in multiple namespace, follow the instructions in the (Link Removed) document and then proceed with the below steps:

1. Execute the following command for DB deployment.

2. Execute the following command for deployment of accelo and deployer.

The following sample commands must be used to push license file, keytab, certs, krb5.conf to pvc.

For HDP or Kafka standalone (ansible deployment), you must generate an SSH key and ensure that you are able to perform password less secure login to the cluster nodes, and then copy the key to the ad-accelo pod as shown below:

3. Execute the following inside the ad-accelo-configure pod and verify the below environments.

4. Configure core in accelo pod by executing the following command:

5. In the pod execute the following command.

The following message displayed:

During deployment, respond to the following questions based on your configurations.

The following block shows sample responses for each of the above questions.

The example of discovered configurations window is displayed below. Additionally, you will find the next set of questions asked.

Notes

• The IP addresses for Hydra is same as the container name. To fix this issue, create a override.yml file in the /data01/acceldata/work/<Cluster_Name>/ location.
• The cluster name for Victoria metrics is obtained from the acceldata_adsample_test.conf. You must replace it in the override.yml file.
• You must add the following environment details:

The content of override.yml file includes the following information:

MongoDB URI Correction

1. Execute the following commands in accelo pod.

2. Replace the MONGO_URI=<Mongo_URI_placeholder> in the /data01/acceldata/config/acceldata<clusterName>.conf file.

For ansible based deployment of hydra (HDP/kafka), fix hydra_server_ ur manually by executing the following code block.

3. Execute the following command.

Deploy Core Services

1. Uncomment the following statements from helmfile/helmfile.yml file, You must perform this step for all core and add-on services. You must retain the comment if you wish to deploy rook-ceph, ad-accelo.

2. The version or build number currently used is containers.tagName=<<release tagname>>. You must execute the following command.

Deploy Add-on Services

1. Execute the following command to deploy the add on services.

You can now access the Pulse UI from the following URL.

FSA Load

You must execute these commands after configuring the add-on services.

1. Connect to ad-accelo-configure, pod and run the following command.

2. Execute the following to configure notifications.

3. Validate if the ad-hydra server container is running and then execute the following command:

Other Configurations

For installing and uninstalling agents in HDP/Kafka (using Ansible-based deployment), after running the sync, you must delete it before proceeding with the "uninstall" tag.

1. Ensure that the /ad-service/ops/helm-controller/helmfile.yaml file has the following content:

Note clusterName is the same name that you provided in the accelo config cluster.

2. Execute the commands in the following block to monitor weave scope.

You can access the weave UI from the http://localhost:4040 UI.

3. Execute the following command to delete the add-ons.

4. To deploy an updated build version, execute the following command. For example, if the current version is 2.1.9-200622 and you wish to upgrade to 2.1.9-200822.