Data Store ADLS

What is ADLS Data Store?

The ADLS Data Store capability in xDP allows you to register and manage connections to your Microsoft Azure Data Lake Storage (ADLS) accounts. By creating a data store, you integrate ADLS as a first-class citizen in the xDP platform, enabling centralized data governance, observability, and access for data processing and analytics workloads. This provides a unified view of your data assets, regardless of where they are stored, simplifying data management and improving data reliability.

Key Concepts

• Data Store: A saved connection configuration within xDP that points to an external data storage system. It acts as a referenceable object for compute jobs, data pipelines, and observability tools, abstracting away the underlying connection details.
• Storage Account: The top-level object in Azure for storing data. An ADLS Gen2 storage account provides a massively scalable and secure data lake for high-performance analytics workloads.
• Container: A logical grouping of data within an Azure Storage Account, similar to a directory in a file system.
• Authentication Method: The mechanism xDP uses to securely connect to your ADLS account. xDP supports two primary methods:
• Storage Account Key: A simple, powerful access key that grants full permissions to the storage account.
• Service Principal (OAuth 2.0): An application identity within Azure Active Directory that can be granted specific, granular permissions to resources, representing a more secure and recommended approach for production environments.

Capabilities

By registering an ADLS Data Store, you can:

• Centralize Data Access: Provide a single, governed point of access to ADLS data for all xDP services and users.
• Enable Data Processing: Use the registered ADLS data store as a source or destination for Spark jobs and other data pipelines orchestrated by xDP.
• Improve Data Governance: Manage and audit connections to critical cloud data assets from a unified control plane.
• Simplify Configuration: Abstract complex connection strings and credentials into a simple, reusable data store object.

Tutorial (Getting Started)

This tutorial guides you through the process of creating your first ADLS Data Store connection in xDP.

Prerequisites

• You have an active Microsoft Azure subscription.
• You have an ADLS Gen2-enabled Storage Account created in Azure.
• You have the required credentials for your chosen authentication method:
• For Storage Account Key authentication: The access key for your storage account.
• For Service Principal authentication: The Tenant ID, Client ID, and Client Secret for an Azure AD application with appropriate permissions (e.g., Storage Blob Data Contributor) on the storage account.
• You have an active Compute Cluster available in xDP.

Your First Workflow

Follow these steps to register a new ADLS data store.

1. From the xDP sidebar, navigate to Platform > Data Store.
2. Click Create Data Store.
3. The Select Cluster step is completed automatically if you have a default cluster. Proceed to the Select Type step.
4. Select ADLS (Azure Data Lake Storage Gen2) from the list of data store types and click Next.
5. On the Basic Details page, enter the connection details for your ADLS account.

• Data Store Name: A unique, descriptive name for this connection (e.g., marketing-events-adls).
• Storage Account Name: The name of your Azure Storage Account.
• Container Name: The specific container within the storage account you want to connect to.
• Container Path (Optional): A specific path within the container to scope the connection.
• Authentication Type: Select your desired authentication method. For this tutorial, we will use Storage Account Key.

6. Enter your Storage Account Key in the provided field.
7. Click Next.
8. Upon success, you are redirected to the Data Stores list, where your new ADLS data store now appears. A confirmation message is displayed at the top of the screen.

How-to Guides

Create an ADLS Data Store using a Service Principal

Using a Service Principal is the recommended authentication method for production environments as it allows for role-based access control and avoids exposing powerful account keys.

1. Navigate to the Data Store page and click Create Data Store.

2. Select ADLS as the data store type and click Next.

3. Fill in the Data Store Name, Storage Account Name, Container Name, and optional Container Path.

4. From the Authentication Type dropdown, select Service Principal (OAuth 2.0).

5. Enter the credentials for your service principal:

   • Tenant ID: Your Azure Active Directory Tenant ID.
   • Client ID: The Application (client) ID of your service principal.
   • Client Secret: The secret value generated for your service principal.

6. Click Next to create the data store.

7. Verification: Confirm that the new data store appears in the list on the main Data Stores page.

Best Practices

• Use Service Principals in Production: For enhanced security and granular control, always use Service Principal authentication for data stores in production environments. This aligns with the principle of least privilege and allows you to revoke access for a specific application without affecting other services.
• Implement Naming Conventions: Adopt a clear and consistent naming convention for your data stores (e.g., <env>-<team>-<purpose>-adls). This makes it easier to identify and manage connections as your platform grows.
• Scope Permissions Tightly: When configuring your Service Principal in Azure, grant it only the minimum required permissions on the specific container or path it needs to access. Avoid granting overly broad roles like Owner or Contributor at the subscription level.
• Manage Credential Rotation: Establish a policy for regularly rotating the credentials used by your data stores (Storage Account Keys or Client Secrets) to minimize the risk of unauthorized access from a compromised credential.