Disaster Recovery Strategy for MongoDB and VictoriaDB

Prerequisites

A minimum of three nodes to enable High Availability are required
MongoDB Version 6 is required for this setup

Steps to Deploy Pulse on New Node

Set the MongoDB URI:

Bash
    
 
export MONGO_URI="<paste previous command output here>"
Copy

Enable MongoDB Encryption:

Bash
    
 
export MONGO_ENCRYPTED=true
Copy

Configure Pulse as a Standalone Node:

Bash
    
 
export PULSE_SA_NODE=true
Copy

Create necessary directory structure:

Bash
    
 
mkdir -p /data01/acceldata/
Copy

Move the Pulse executable to the designated directory:

Bash
    
 
mv accelo.linux /data01/acceldata/accelo
Copy

Navigate to the Pulse directory:

Bash
    
 
cd /data01/acceldata/
Copy

Initialize Pulse:

Bash
    
 
./accelo init
Copy

Source the environment variables:

Bash
    
 
source /etc/profile.d/ad.sh
Copy

Reinitialize Pulse:

Bash
    
 
accelo init
Copy

Retrieve Pulse information:

Bash
    
 
accelo info
Copy

Bash
    
 
accelo login docker
Copy

Add Host Entries

Add the following entries to the /etc/hosts file of all the MongoDB nodes, including the Pulse server:

Bash
    
 
<Pulse-Server-IP> ad-db<Second-Node-IP> ad-db2<Third-Node-IP> ad-db3
Copy

Set New MongoDB Nodes

Retrieve the accelo binary and initiate it with the same version tag as the Pulse version on the primary node.

Bash
    
 
accelo init
Copy

Docker
    
 
docker network create ad-default
Copy

Create a new Docker network:

Docker
    
 
docker network create ad-default
Copy

Manually download the Mongo image:

Docker
    
 
docker pull 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>
Copy

Configuring the Primary Node (Pulse Server)

Edit Configuration File: In the config/docker/ad-core.yaml file, add the following new fields to the ad-db service section:

Add the following fields to the environment block:

Bash
    
 
ENABLE_MONGODB_RS=trueMONGODB_RS_NAME=addb0IS_MONGODB_PRIMARY=true
Copy

Add the following fields to the volumes block:

Bash
    
 
/etc/localtime:/etc/localtime:ro/root/acceldata/data/db:/data/db/root/acceldata/config/db/mongokey:/mongokeyAdd the following fields to the extra_hosts block:
Copy

Add the following fields to the extra_hosts block:

Bash
    
 
ad-db2:<Second-Node-IP>ad-db3:<Third-Node-IP>
Copy

Run Secondary Data Replica: Replace <Pulse-Server-IP>, <Third-Node-IP>, and <PULSE-VERSION> with the appropriate values and execute:

Bash
    
docker run -d --name ad-db3_default -p 27017:27017 -e MONGO_DATA_DIR=/data/db -e MONGO_LOG_DIR=/dev/null -e ENABLE_MONGODB_RS=true -e MONGODB_RS_NAME=addb0 -v /data01/acceldata/config/db/mongokey:/mongokey -v /etc/localtime:/etc/localtime:ro -v /data01/acceldata/data/db:/data/db --restart=unless-stopped --net=ad-default --network-alias=ad-db3 --add-host ad-db:<Pulse-Server-IP> --add-host ad-db2:<Second-Node-IP> 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>
Copy

Run Third Data Replica: Replace <Pulse-Server-IP>, <Second-Node-IP>, and <PULSE-VERSION> with the appropriate values and execute:

Bash
    
docker run -d --name ad-db3_default -p 27017:27017 -e MONGO_DATA_DIR=/data/db -e MONGO_LOG_DIR=/dev/null -e ENABLE_MONGODB_RS=true -e MONGODB_RS_NAME=addb0 -v /data01/acceldata/config/db/mongokey:/mongokey -v /etc/localtime:/etc/localtime:ro -v /data01/acceldata/data/db:/data/db --restart=unless-stopped --net=ad-default --network-alias=ad-db3 --add-host ad-db:<Pulse-Server-IP> --add-host ad-db2:<Second-Node-IP> 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>
Copy

Login to Primary Node's MongoDB Shell & Add Replica Members:

Bash
    
chown 1000:1000 /data01/acceldata/config/db/mongokeydocker exec -it ad-db_default bashmongosh mongodb://admin:ACCELROOT_01082018@localhost:27017/admindb.adminCommand({"replSetReconfig":{_id: "addb0",version: 2,members: [{ _id : 0, host : "ad-db:27017" },{ _id : 1, host : "ad-db2:27017" },{ _id : 2, host : "ad-db3:27017" }]}})
Copy

Check Replica Status: Run rs.status() to check the status of the replica.
Wait for Stable State: Wait for the MongoDB cluster to stabilize.
Update Mongodb Encrypted URI: Replace the MongoDB encrypted URI in all YAML files under config/docker/ and acceldata_acceluster.conf file with the provided value.

Unencrypted string for reference:

Bash
    
 
mongodb://accel:ACCELUSER_01082018@ad-db:27017,ad-db2:27017,ad-db3:27017/admin?replicaSet=addb0
Copy

VictoriaDB Setup for Backup and Restoration

This is specifically aimed at addressing disk failures; hence, the destination path will reside on a distinct mounted disk.

To setup Victoria database for backup and restore, perform the following:

Navigate to the Acceldata directory.
Fetch the binaries you want to backup and restore using the below command:

Bash
    
 
wget "https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.97.1/vmutils-linux-amd64-v1.97.1.tar.gz "
Copy

Extract the contents of the above tar file:

Bash
    
 
tar -xvf vmutils-linux-amd64-v1.97.1.tar.gz
Copy

Remove unnecessary binaries, except vmbackup and vmrestore.
Execute the shell script:

Bash
    
 
source /etc/profile.d/ad.sh
Copy

Create a new directory for vmdbbackup as below:

Bash
    
 
mkdir $AcceloHome/data/vmdbbackupchown 1000:1000 $AcceloHome/data/vmdbbackupsu - $(id -un 1000)
Copy

Open up access to the ad-vmstorage port by modifying the ad-core.yml file as shown below.

Bash
    
 
ad-vmstorage:    image: ad-vmstorage    container_name: ""    environment:    - ARGS=--storageDataPath=/storage    volumes:    - /etc/localtime:/etc/localtime:ro    - /data01/acceldata/data/vmdb:/storage    ulimits: {}    ports:    - 19099:8482    depends_on: []    opts: {}    restart: always    extra_hosts: []    network_alias: []
Copy

Restart vmstorage container using:

Bash
    
 
accelo restart ad-vmstorage
Copy

Execute the following command. This command initiates an incremental backup process. Incremental backups are conducted when the destination (-dst) parameter refers to an existing backup directory. After the initial backup is completed, subsequent backups will always target this existing backup directory.

Bash
    
vmbackup-prod -storageDataPath=/data01/acceldata/data/vmdb -snapshot.createURL=http://localhost:19099/snapshot/create -dst=fs:///data01/acceldata/data/vmdbbackup
Copy

Configure the crontab to execute this task once daily, preferably in the morning:

Bash
    
crontab -u user -e 0 1 * * * /data01/acceldata/vmbackup-prod -storageDataPath=/data01/acceldata/data/vmdb -snapshot.createURL=http://localhost:19099/snapshot/create -dst=fs:///data01/acceldata/data/vmdbbackup
Copy

Steps to Restore

To restore from backup, perform the following:

Stop the vmstorage container, if it already running, and run the restore command:

Bash
    
 
docker stop ad-vmstorage_default vmrestore-prod -src=fs:///data01/acceldata/data/vmdbbackup -storageDataPath=/data01/acceldata/data/vmdb
Copy

Run accelo deploy core
Check the logs of the container using docker logs -f ad-vmstorage_default

Troubleshooting

If the backup process is slow, consider adjusting the value of the -concurrency flag to a higher setting. This will boost the number of simultaneous workers responsible for uploading data to the backup storage.
If vmrestore or vmbackup encounters an interruption due to a temporary error, simply restart it using the same arguments. The process will resume from where it left off, continuing the restore or backup operation.

Last updated on

Was this page helpful?