Global Storage Setup for NextGen Dataplane

NextGen Dataplane Global Storage Setup

Global storage is required for storing data or results in a variety of file formats across multiple use cases.

This article gives step-by-step directions for configuring global storage for the NextGen Dataplane using various storage types.

Supported Storage Types

Google Bucket Storage
AWS Storage
Azure Storage
Hadoop File System (HDFS/MAPRFS)
Local to the Pod

Standard Location for Global Storage Configuration

The configuration file for global storage is located at /opt/acceldata/globalstorage.json.

Steps for Configuration:

ADOC contains global-storage secrets with the below variables. As part of Data Plane V2, the global storage configuration is stored as a Kubernetes secret named global-storage. Use the JSON configuration below, encode it with base64 before applying it as a secret:

JSON
    
 
{    "MEASURE_RESULT_FS_TYPE": "s3",    "MEASURE_RESULT_FS_HDFS_DIRECTORY": "/tmp/ad/torch_results",    "MEASURE_RESULT_FS_S3A_BUCKET": "<bucket for global storage>",    "MEASURE_RESULT_FS_S3A_ACCESS_KEY": "",    "MEASURE_RESULT_FS_S3A_SECRET_KEY": "",    "MEASURE_RESULT_FS_S3A_REGION": "<region of bucket>",    "MEASURE_RESULT_FS_S3A_MODE": "IAM_ROLES_FOR_SA_BASED",    "MEASURE_RESULT_FS_S3A_ACCESS_INSTANCE_PROFILE_ARN": "",    "MEASURE_RESULT_FS_GCS_BUCKET": "",    "MEASURE_RESULT_FS_GCS_KEY_FILE": "/opt/acceldata/gcp_cred.json",    "MEASURE_RESULT_FS_GCS_PROJECT_ID": "",    "MEASURE_RESULT_FS_GCS_CLIENT_EMAIL": "",    "MEASURE_RESULT_FS_GCS_PRIVATE_KEY": "",    "MEASURE_RESULT_FS_GCS_PRIVATE_KEY_ID": "",    "MEASURE_RESULT_FS_ADLS_GENERATION": "GEN_2",    "MEASURE_RESULT_FS_ADLS_STORAGE_ACCOUNT_NAME": "",    "MEASURE_RESULT_FS_ADLS_STORAGE_CONTAINER_NAME": "",    "MEASURE_RESULT_FS_ADLS_STORAGE_ACCESS_KEY": "",    "MEASURE_RESULT_FS_ADLS_STORAGE_CLIENT_ID": "",    "MEASURE_RESULT_FS_ADLS_STORAGE_CLIENT_SECRET": "",    "MEASURE_RESULT_FS_ADLS_STORAGE_TENANT_ID": "",    "MEASURE_RESULT_FS_GCS_MODE": "SERVICE_ACCOUNT_KEY_FILE_BASED"}
Copy

Leave the following two fields as blank:

MEASURE_RESULT_FS_S3A_ACCESS_KEY": "",

"MEASURE_RESULT_FS_S3A_SECRET_KEY": "",

Prepare or update the configuration for the selected storage type.

Google Bucket

AWS S3

Azure ADLS

HDFS/MAPRFS

Local Storage to Pod

Ensure MEASURE_RESULT_FS_TYPEis set to gcs.

JSON
    
xxxxxxxxxx
 
MEASURE_RESULT_FS_TYPE  : gcs
Copy

Provide the bucket name, project ID, client email, private key, and private key ID.

JSON
    
xxxxxxxxxx
 
MEASURE_RESULT_FS_GCS_BUCKET :  Bucket name to store the resultsMEASURE_RESULT_FS_GCS_MODE: should be SERVICE_ACCOUNT_KEY_FILE_BASED
Copy

GCP credential file-based setup

JSON
    
xxxxxxxxxx
 
MEASURE_RESULT_FS_GCS_KEY_FILE: /opt/acceldata/gcp_cred.jsonNote - This is the standard location.You don't need to update.
Copy

Alternatively directly accessing the below configuration; generally, the config JSON provides the same details.

JSON
    
xxxxxxxxxx
 
MEASURE_RESULT_FS_GCS_PROJECT_ID : Project Id​MEASURE_RESULT_FS_GCS_CLIENT_EMAIL : Service account Email​MEASURE_RESULT_FS_GCS_PRIVATE_KEY : Private key which we can get in credential file​MEASURE_RESULT_FS_GCS_PRIVATE_KEY_ID : Private key id , same we can get in credential file
Copy

You can either Edit Secret and update with base64 content for key globalstorage.json and value as base64 content,

JSON
    
xxxxxxxxxx
 
kubectl edit secret global-storage -n <namespace>
Copy

When using GCP as a bucket, ADOC have a consistent place for file-driven credentials.

JSON
    
xxxxxxxxxx
 
/opt/acceldata/gcp_cred.json
Copy

Update the content.

Edit Secret gcp-cred and update with base64 content for key gcp_cred.json and value in base64 format.

JSON
    
xxxxxxxxxx
 
kubectl edit secrets gcp-cred -n <namespace>
Copy

Last updated on

Was this page helpful?