Disaster Recovery Strategy for MongoDB and VictoriaDB

Prerequisites

• A minimum of three nodes to enable High Availability are required
• MongoDB Version 6 is required for this setup

Steps to Deploy Pulse on New Node

1. Set the MongoDB URI:

export MONGO_URI="<paste previous command output here>"

2. Enable MongoDB Encryption:

export MONGO_ENCRYPTED=true

3. Configure Pulse as a Standalone Node:

export PULSE_SA_NODE=true

4. Create necessary directory structure:

mkdir -p /data01/acceldata/

5. Move the Pulse executable to the designated directory:

mv accelo.linux /data01/acceldata/accelo

6. Navigate to the Pulse directory:

cd /data01/acceldata/

7. Initialize Pulse:

./accelo init

8. Source the environment variables:

source /etc/profile.d/ad.sh

9. Reinitialize Pulse:

accelo init

10. Retrieve Pulse information:

accelo info

11. Log in to Pulse Docker:

accelo login docker

Add Host Entries

Add the following entries to the /etc/hosts file of all the MongoDB nodes, including the Pulse server:

<Pulse-Server-IP> ad-db

<Second-Node-IP> ad-db2

<Third-Node-IP> ad-db3

Set New MongoDB Nodes

1. Retrieve the accelo binary and initiate it with the same version tag as the Pulse version on the primary node.

accelo init

2. Login to the Pulse Registry:

docker network create ad-default

3. Create a new Docker network:

docker network create ad-default

4. Manually download the Mongo image:

docker pull 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>

Configuring the Primary Node (Pulse Server)

1. Edit Configuration File: In the config/docker/ad-core.yaml file, add the following new fields to the ad-db service section:

• Add the following fields to the environment block:

ENABLE_MONGODB_RS=true

MONGODB_RS_NAME=addb0

IS_MONGODB_PRIMARY=true

• Add the following fields to the volumes block:

/etc/localtime:/etc/localtime:ro

/root/acceldata/data/db:/data/db

/root/acceldata/config/db/mongokey:/mongokey

Add the following fields to the extra_hosts block:

• Add the following fields to the extra_hosts block:

ad-db2:<Second-Node-IP>

ad-db3:<Third-Node-IP>

2. Run Secondary Data Replica: Replace <Pulse-Server-IP>, <Third-Node-IP>, and <PULSE-VERSION> with the appropriate values and execute:

docker run -d --name ad-db3_default -p 27017:27017 -e MONGO_DATA_DIR=/data/db -e MONGO_LOG_DIR=/dev/null -e ENABLE_MONGODB_RS=true -e MONGODB_RS_NAME=addb0 -v /data01/acceldata/config/db/mongokey:/mongokey -v /etc/localtime:/etc/localtime:ro -v /data01/acceldata/data/db:/data/db --restart=unless-stopped --net=ad-default --network-alias=ad-db3 --add-host ad-db:<Pulse-Server-IP> --add-host ad-db2:<Second-Node-IP> 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>

3. Run Third Data Replica: Replace <Pulse-Server-IP>, <Second-Node-IP>, and <PULSE-VERSION> with the appropriate values and execute:

docker run -d --name ad-db3_default -p 27017:27017 -e MONGO_DATA_DIR=/data/db -e MONGO_LOG_DIR=/dev/null -e ENABLE_MONGODB_RS=true -e MONGODB_RS_NAME=addb0 -v /data01/acceldata/config/db/mongokey:/mongokey -v /etc/localtime:/etc/localtime:ro -v /data01/acceldata/data/db:/data/db --restart=unless-stopped --net=ad-default --network-alias=ad-db3 --add-host ad-db:<Pulse-Server-IP> --add-host ad-db2:<Second-Node-IP> 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>

4. Login to Primary Node's MongoDB Shell & Add Replica Members:

chown 1000:1000 /data01/acceldata/config/db/mongokey

docker exec -it ad-db_default bash

mongosh mongodb://admin:ACCELROOT_01082018@localhost:27017/admin

db.adminCommand({"replSetReconfig":{_id: "addb0",version: 2,members: [{ _id : 0, host : "ad-db:27017" },{ _id : 1, host : "ad-db2:27017" },{ _id : 2, host : "ad-db3:27017" }]}})

5. Check Replica Status: Run rs.status() to check the status of the replica.
6. Wait for Stable State: Wait for the MongoDB cluster to stabilize.
7. Update Mongodb Encrypted URI: Replace the MongoDB encrypted URI in all YAML files under config/docker/ and acceldata_acceluster.conf file with the provided value.

Unencrypted string for reference:

mongodb://accel:ACCELUSER_01082018@ad-db:27017,ad-db2:27017,ad-db3:27017/admin?replicaSet=addb0

VictoriaDB Setup for Backup and Restoration

To setup Victoria database for backup and restore, perform the following:

1. Navigate to the Acceldata directory.
2. Fetch the binaries you want to backup and restore using the below command:

wget "https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.97.1/vmutils-linux-amd64-v1.97.1.tar.gz "

3. Extract the contents of the above tar file:

tar -xvf vmutils-linux-amd64-v1.97.1.tar.gz

4. Remove unnecessary binaries, except vmbackup and vmrestore.
5. Execute the shell script:

source /etc/profile.d/ad.sh

6. Create a new directory for vmdbbackup as below:

mkdir $AcceloHome/data/vmdbbackup

chown 1000:1000 $AcceloHome/data/vmdbbackup

su - $(id -un 1000)

7. Open up access to the ad-vmstorage port by modifying the ad-core.yml file as shown below.

ad-vmstorage:

    image: ad-vmstorage

    container_name: ""

    environment:

    - ARGS=--storageDataPath=/storage

    volumes:

    - /etc/localtime:/etc/localtime:ro

    - /data01/acceldata/data/vmdb:/storage

    ulimits: {}

    ports:

    - 19099:8482

    depends_on: []

    opts: {}

    restart: always

    extra_hosts: []

    network_alias: []

8. Restart vmstorage container using:

accelo restart ad-vmstorage

9. Execute the following command. This command initiates an incremental backup process. Incremental backups are conducted when the destination (-dst) parameter refers to an existing backup directory. After the initial backup is completed, subsequent backups will always target this existing backup directory.

vmbackup-prod -storageDataPath=/data01/acceldata/data/vmdb -snapshot.createURL=http://localhost:19099/snapshot/create -dst=fs:///data01/acceldata/data/vmdbbackup

10. Configure the crontab to execute this task once daily, preferably in the morning:

crontab -u user -e 

0 1 * * * /data01/acceldata/vmbackup-prod -storageDataPath=/data01/acceldata/data/vmdb -snapshot.createURL=http://localhost:19099/snapshot/create -dst=fs:///data01/acceldata/data/vmdbbackup

Steps to Restore

To restore from backup, perform the following:

1. Stop the vmstorage container, if it already running, and run the restore command:

docker stop ad-vmstorage_default 

vmrestore-prod -src=fs:///data01/acceldata/data/vmdbbackup -storageDataPath=/data01/acceldata/data/vmdb

2. Run accelo deploy core
3. Check the logs of the container using docker logs -f ad-vmstorage_default

Troubleshooting

• If the backup process is slow, consider adjusting the value of the -concurrency flag to a higher setting. This will boost the number of simultaneous workers responsible for uploading data to the backup storage.
• If vmrestore or vmbackup encounters an interruption due to a temporary error, simply restart it using the same arguments. The process will resume from where it left off, continuing the restore or backup operation.