In the Data Sources window, click the Create Data Source button and select the Hive option to create a data source. The user can either Create New Connection, or Use Existing Connection to connect to the Hive data source.
Create New Connection
To create a new connection, enter the following parameters:
| Parameter | Description |
|---|---|
| Connection Name | Specify the name for the connection. It is a required field which is not case sensitive and must be unique in the domain. It should not exceed 128 characters and can contain special characters. |
| Description | Describe the purpose of the connection. The description cannot exceed 4000 characters. |
| Use Secret Manager | Click the checkbox to use secret manager and then select a secret manager identifier. In the secret field, provide the key name which you have stored in AWS Secrets Manager that contains the actual password value. For more information, see Use Secret Manager. |
| JDBC URL | Specify the Java Database Connectivity (JDBC) URL which is used to locate the database schema. The URL uses the following format:
|
| JDBC Username | Specify the username to connect to the Hive database. The user must have access to all the hive meta-store tables. |
| JDBC Password | Specify the login password. |
Click Test Connection to check if the connection created is working.
Only a single Hive connection can be configured per analytics service. While deploying the application, the hive-site.xml, hdfs-site.xml and the core-site.xml files must be provided in the analytics configuration section. If the Hadoop cluster is Kerberos enabled, HDFS headless keytab and principal must be provided.
Use Secret Manager
AWS Secrets Manager is a secrets management service that helps you protect access to your applications, services, and IT resources. This service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. We are enabling torch to make use of AWS Secrets Manager to store your secrets. Acceldata provides the option to store integration credentials in your AWS account using Secrets Manager.
Follow the below steps to create secrets in AWS:
- Login to your AWS account.
- Search for Secrets Manager.
- Click Store a new secret.
- Select Other type of secrets and add a key with a value representing the actual secret value like a password of a database user and click Next.
- Give the secret a meaningful name, description and add any tags. Then click Next.
How do I enable Torch to read from AWS Secret Manager?
While Deploying Data Plane or Complete Installation, you are provided with an option to configure secrets manager configuration
- Under Secret Keys Configuration > Click the Enable Key Management checkbox.
- Upload the following secret manager configuration file in JSON format:
[ { "name": "<Name to identify the secret manager to use in torch>", "type": "AWS_SECRETS_MANAGER", "details": { "secretName": "<Secret Name which is created in AWS Secret Manager>", "accessKey": "<AWS access Key>", "secretKey": "<AWS Secret Key>", "region": "<AWS Region where secret is created>" } }]The above configuration file is a JSON array, where each element represents a secret configuration. There are 2 ways in which Torch can authenticate itself to AWS Secrets Manager.
i) Providing the accessKey details of an IAM User who has the permission to read the Secret
ii) Leaving the accessKey and secretKey fields as empty and Torch assumes that there is a IAM Role attached to the NodeGroup of EKS Cluster which has the permission to read the Secret(EC2InstanceProfile).
Refer to the following document for IAM Policies to be attached to the NodeGroups https://docs.aws.amazon.com/mediaconnect/latest/ug/iam-policy-examples-asm-secrets.html.
Use Existing Connection
To use an existing connection, click Use Existing Connection tab to select an existing connection from the drop-down list.
Data Source Configuration
Enter the following parameters for the Data Source Configuration:
| Parameter | Description |
|---|---|
| Name | Name for the data source configuration |
| Description | Description for the data source configuration |
| Databases | Select a database from the drop-down list. Only the selected databases will be crawled. |
Click Next.
Crawler Execution Schedule
Click Use automated execution checkbox to schedule a time for Torch to run metadata crawlers to fetch meta information of the data source. To schedule, select any tag like minute, hour, day, week, month, or year.
Click Save button.