AWS Glue
In the Data Sources window, click the Create Data Source button and select the AWS Glue option to create a data source. The user can either Create New Connection or Use Existing Connection to connect to the AWS data source.
Create New Connection
Amazon Web Services (AWS) connection type enables AWS integrations. To create a new connection, enter the following parameters:
| Parameter | Description |
|---|---|
| Connection Name | Name of the connection is a required field that is not case sensitive and must be unique in the domain. It should not exceed 128 characters and can contain special characters as well. |
| Description | Describe the purpose of the connection. The description cannot exceed 4000 characters. |
| Use Secret Manager | Click the checkbox to use secret manager and then select a secret manager identifier. In the secret field, provide the key name which you have stored in AWS Secrets Manager that contains the actual password value. For more information, see Use Secret Manager. |
| AWS Access Key | Fill in the AWS access key ID. |
| AWS Secret Key | Enter the AWS secret access key. |
| AWS region | Enter the AWS region for the glue data lake. (example: us-east-1) |
Click Test Connection to check if the connection created is working.
Use Secret Manager
AWS Secrets Manager is a secrets management service that helps you protect access to your applications, services, and IT resources. This service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. We are enabling torch to make use of AWS Secrets Manager to store your secrets. Acceldata provides the option to store integration credentials in your AWS account using Secrets Manager.
Follow the below steps to create secrets in AWS:
- Login to your AWS account.
- Search for Secrets Manager.
- Click Store a new secret.
- Select Other type of secrets and add a key with a value representing the actual secret value like a password of a database user and click Next.
- Give the secret a meaningful name, description and add any tags. Then click Next.
How do I enable Torch to read from AWS Secret Manager?
While Deploying Data Plane or Complete Installation, you are provided with an option to configure secrets manager configuration
- Under Secret Keys Configuration > Click the Enable Key Management checkbox.
- Upload the following secret manager configuration file in JSON format:
[ { "name": "<Name to identify the secret manager to use in torch>", "type": "AWS_SECRETS_MANAGER", "details": { "secretName": "<Secret Name which is created in AWS Secret Manager>", "accessKey": "<AWS access Key>", "secretKey": "<AWS Secret Key>", "region": "<AWS Region where secret is created>" } }]The above configuration file is a JSON array, where each element represents a secret configuration. There are 2 ways in which Torch can authenticate itself to AWS Secrets Manager.
i) Providing the accessKey details of an IAM User who has the permission to read the Secret
ii) Leaving the accessKey and secretKey fields as empty and Torch assumes that there is a IAM Role attached to the NodeGroup of EKS Cluster which has the permission to read the Secret(EC2InstanceProfile).
Refer to the following document for IAM Policies to be attached to the NodeGroups https://docs.aws.amazon.com/mediaconnect/latest/ug/iam-policy-examples-asm-secrets.html.
Use Existing Connection
To use an existing connection, click Use Existing Connection tab to select an existing connection from the drop-down list.
Data Source Configuration
Specify a name and description for the data source configuration. Click Next.
Crawler Execution Schedule
Click Use automated execution checkbox to schedule a time for Torch to run metadata crawlers to fetch meta information of the data source. To schedule, select any tag like minute, hour, day, week, month, or year.
Click Save button. The data source is saved.
Required Permissions for AWS Glue
While creating the AWS Identity and Access Management (IAM) user, please include permissions by attaching the following policies to demonstrate the required allowed actions and resources for the user:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "glue:GetTables", "s3:AbortMultipartUpload", "glue:GetDatabases", "s3:GetBucketLocation", "glue:GetTable" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<data-bucket-path>" ] }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": [ "arn:aws:s3:::<data-bucket-path>/*" ] } ]}