Databricks

Databricks is a cloud-based big data processing platform. You can use Databricks to easily configure and deploy data processing clusters with a few clicks.

This document explains how to add Databricks on Azure as a data source in the Acceldata Data Observability Cloud (ADOC). Once you add Databricks as a data source, you can monitor your cluster health, notebook status, and costs incurred. You can view all this data in visual format. You can also create alerts on various Databricks entities.

Take a look at this video which explains the process of adding Databricks as a data source.

Steps to Add Databricks as a Data Source

To add Databricks as a data source in ADOC, follow these steps:

Prerequisites

Before you begin, ensure you have:

• An account with the necessary permissions for your cloud provider(Azure or AWS).
• Access to the Azure Portal.
• Access to your Databricks workspace.
• Access to the ADOC platform.
• Relevant credentials depending on your cloud provider:
  • Azure: Subscription ID, Tenant ID, Resource Group, Managed Resource Group.
  • AWS: IAM user credentials (Access Key, Secret Key) with necessary Cost Explorer and Pricing permissions.

Azure

1. Creating a Service Principal

A Service Principal is required to enable API access and authenticate with Databricks.

1. Go to Azure Portal: Navigate to https://portal.azure.com/ and log in.

2. Access Microsoft Entra ID (Azure Active Directory): From the left-hand menu, select Microsoft Entra ID.

3. Register a New Application

   1. In the Manage section, select App registrations.

   2. Click on New registration.

   3. Provide the following details:

      • Name: e.g., ADOC Databricks Integration.
      • Supported account types: Accounts in this organizational directory only.

   4. Click Register.

4. Create a Client Secret

   1. In the application's Overview page, select Certificates & secrets from the Manage section.
   2. Click on New client secret.
   3. Enter a description and select an expiration period.
   4. Click Add.
   5. Important: Copy the Value of the client secret. This is your Azure Client Secret Value. Save it securely, as it is visible only once.

2. Creating a Custom Role and Assign It to the Service Principal

Access Control (IAM) in Azure

1. Navigate to Your Subscription: Select Subscriptions from the left-hand menu and choose your subscription.

2. Access Control (IAM): In the subscription menu, click Access control (IAM).

3. Create a Custom Role

   1. Click Add and select Add custom role.
   2. Provide a Custom role name (e.g., Cost Management Reader).
   3. In the Basics tab, enter a description if desired.

4. Add Permissions

   1. Go to the Permissions tab.

   2. Click Add permissions.

   3. In the Add permissions pane:

      • Provider: Select Microsoft.CostManagement.
      • Permission: Search for and add the following permissions:
        • Microsoft.CostManagement/exports/read
        • Microsoft.CostManagement/query/read

   4. Click Add.

5. Review and Create

   1. Click Review + create.
   2. Review the details and click Create.

Assign the Custom Role to the Service Principal

1. Return to Access Control (IAM): Ensure you're in the Access control (IAM) section of your subscription.

2. Add Role Assignment: Click Add and select Add role assignment.

3. Assign the Custom Role

   1. In the Role dropdown, select the custom role you created.
   2. Click Next.

4. Select Members

   1. Under Members, choose User, group, or service principal.
   2. Click Select members.
   3. Search for your Service Principal by name.
   4. Select it and click Select.

5. Review and Assign

   1. Click Review + assign.
   2. Confirm and click Assign.

3. Adding the Service Principal to the Databricks Workspace

1. Log In to Databricks Workspace: Navigate to your Databricks workspace URL.

2. Access Service Principals

   1. Click on the Settings icon (⚙️) in the lower-left corner.
   2. Select Admin Console.
   3. Go to the Service Principals tab.

3. Add Service Principal

   1. Click Add Service Principal.
   2. Select Microsoft Entra ID managed.
   3. Enter the Application (client) ID of your Service Principal.
   4. Provide a Display name (can be any meaningful name).
   5. Click Add.

4. Provide Workspace Admin Access to the Service Principal

1. Access the Admin Group

   1. In the Admin Console, select the Groups tab.
   2. Click on the admins group.

2. Add the Service Principal to the Admin Group

   1. Click Add Members.
   2. Search for your Service Principal.

Select it and click Confirm.

5. Retrieve Azure Tenant ID and Other Credentials

You will need the following information:

AWS

1. Creating an IAM User with Cost Explorer and Pricing Permissions

To retrieve cost data from AWS, you need an IAM user with the required permissions:

1. Log in to the AWS Management Console.
2. Create an IAM user with the following permissions (Cost Explorer and Pricing):

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Sid": "VisualEditor0",

      "Effect": "Allow",

      "Action": [

        "ce:GetCostAndUsage",

        "ce:GetReservationPurchaseRecommendation",

        "ce:GetPreferences",

        "ce:ListSavingsPlansPurchaseRecommendationGeneration",

        "ce:ListTagsForResource",

        "ce:GetReservationUtilization",

        "ce:GetCostCategories",

        "ce:GetSavingsPlansPurchaseRecommendation",

        "ce:GetSavingsPlansUtilizationDetails",

        "ce:GetDimensionValues",

        "ce:GetAnomalySubscriptions",

        "ce:DescribeReport",

        "ce:GetReservationCoverage",

        "ce:GetAnomalyMonitors",

        "ce:GetUsageForecast",

        "ce:DescribeNotificationSubscription",

        "ce:DescribeCostCategoryDefinition",

        "ce:GetRightsizingRecommendation",

        "ce:GetSavingsPlansUtilization",

        "ce:GetAnomalies",

        "ce:ListCostCategoryDefinitions",

        "ce:GetCostForecast",

        "ce:GetApproximateUsageRecords",

        "ce:GetCostAndUsageWithResources",

        "ce:ListCostAllocationTags",

        "ce:GetSavingsPlanPurchaseRecommendationDetails",

        "ce:GetSavingsPlansCoverage",

        "ce:GetConsoleActionSetEnforced",

        "ce:GetTags",

        "ce:ListCostAllocationTagBackfillHistory",

        "pricing:DescribeServices",

        "pricing:GetAttributeValues",

        "pricing:GetProducts",

        "pricing:*"

      ],

      "Resource": "*"

    }

  ]

}

3. This IAM user will provide the AWS Access Key ID and Secret Access Key for cost retrieval.

2. Creating a Personal Access Token in Databricks

For AWS Databricks integration, you must use a personal access token to connect to system tables (for billing data and metadata):

1. Log in to your Databricks workspace.
2. Create a Personal Access Token if you do not already have one.
3. If you have a service principal, you can also create a personal access token from that principal.
4. Keep this token secure, as it will be required during ADOC data source registration.

3. Configuring the ADOC Platform

1. Log in to ADOC.
2. Click Register on the left pane, then Add Data Source.
3. Select Databricks.
4. Enter Basic Details:

5. Enter Databricks Connection Details for AWS:

Click Test Connection. If connected, click Next.

4. Setting Up Global Init Script

• In the final step of data source registration, you can enable setting up a global init script.
• By enabling this option, an init script is deployed from Acceldata to your Databricks workspace environment.
• The init script contains details of agent binaries deployed on the customer’s Databricks environment for pushing spark-related compute metrics to ADOC.

5. Providing DBU Values for Estimated Cost Calculation

Provide DBU values (such as Jobs Compute, Jobs Photon Compute, DLT, All-Purpose Photon Compute) as per the Databricks contract. These values are used for cost estimation of the current day based on DBUs.

6. Configuring AWS Cloud Provider Details and Cost Retrieval

1. Cloud Provider Cost Discount Percentage: Enter the discount percentage provided by AWS to the customer.

2. Cloud Provider: Select AWS.

3. Cloud Region: Provide the AWS region where your Databricks workspace is deployed. To find the region:

   1. Go to your Databricks workspace.
   2. Click on the workspace name at the top-right corner.
   3. The region will be displayed.

4. Cost Fetch Method: Select API.

5. AWS Access Key ID and Secret: Enter the credentials of the IAM user created in Step 1. This allows ADOC to retrieve cloud vendor billing information.

6. Click Submit to complete the setup.

Also see, Create IAM user.

Configuring the ADOC Platform

Onboarding a New Datasource or Updating an Existing One

1. Click Register from the left pane.
2. Click Add Data Source.
3. Select the Databricks Data Source. The Databricks Data Source basic Details page is displayed.

4. Enter a name for the data source in the Data Source name field.
5. (Optional) Enter a description for the Data Source in the Description field.
6. (Optional) Enable the Compute capability by switching on the Compute toggle switch.
7. (Optional) Enable the Data Reliability capability by switching on the Data Reliability toggle switch.

8. Select a Data Plane from the Select Data Plane drop-down menu.

To create a new Data Plane, click Setup Dataplane.

9. Click Next. The Databricks Connection Details page is displayed.
10. Enter the name for your Databricks workspace in the Workspace Name field.
11. Enter the URL of your Databricks account in the Databricks URL field. To learn more about workspaces, refer this Databricks document.
12. Enter the access token for your Databricks account in the Token field or enable the Use Service Principal toggle button and provide the below details. To learn more about Tokens, refer this Databricks document.
    1. Select one of the following cloud provider for your Databricks workspace from the drop-down: AWS or Azure.
    2. Provide the Service Principal Client Id to connect to your Databricks workspace.
    3. Provide the Service Principal Client Secret key.
    4. Provide the Service Principal Tenant Id or Account Id for the cloud provider selected in step 12 (a).

13. If you enabled the Data Reliability capability, you must enter the JDBC URL in the JDBC URL field. This field is displayed only if you enabled Data Reliability in step 7.
14. Choose the Dataplane Engine—either Spark or Pushdown—for profiling, data quality checks, and SQL operations. Note Token-based authentication is currently the only supported method for the Pushdown engine on AWS and Azure. For more information, see Pushdown Data Engine.
15. Click Test Connection to validate your credentials and establish a connection with your Databricks account.

If the connection is successful, a Connected message is displayed. If there is an error, you must verify the Databricks details provided and enter the correct details.

16. Click Next to proceed to the Observability Set Up page.

Observability Set Up

The Observability Set Up page allows you to configure Compute and Data Reliability capabilities. These sections are active only if you enabled them on the Databricks Data Source Basic Details page.

This section includes the following panels:

Compute

Provide the following details to calculate compute costs for your Databricks workspace in ADOC.

If you want to retrieve actual cost values, then toggle the Enable Azure Actual Cost button and provide the following details.

Make sure to generate a service principal in your Azure portal so that you can input the necessary parameter values listed below.

Data Reliability

Enter the following details to set up Data Reliability:

• Unity Catalog Enabled: Turn on this toggle switch to enable Unity Catalog in Databricks.
• JDBC Records Fetch Size: Enter the number of records to be fetched from JDBC.
• Enable Crawler Execution Schedule: Turn on this toggle switch to select a time tag and time zone to schedule the execution of crawlers for Data Reliability.

Click the Submit button.

Databricks is now added as a Data Source. You can choose to crawl your Databricks account now or later.

Authentication Mechanism: Service Principals

ADOC introduces an improved authentication process for Databricks by transitioning from personal access tokens to a more secure and managed Service Principal approach.

Configuring Service Principals

1. Establish a Service Principal: Follow the steps outlined in the Creating a Service Principal section above.
2. Assign Roles and Permissions: Configure the service principal's roles and permissions at the workspace level to ensure access to the relevant resources. This involves creating a custom role and assigning it to the service principal as described earlier.
3. Add the Service Principal to Databricks Workspace: Add the service principal to your Databricks workspace by following the steps in the Adding the Service Principal to the Databricks Workspace section.
4. Provide Workspace Admin Access: Provide admin access to the service principal in your Databricks workspace as described in the Provide Workspace Admin Access to the Service Principal section.
5. Configure in ADOC: When adding or editing a Databricks data source in ADOC, select the Use Service Principal option and provide the service principal details.

All data activities, including data crawling, data profiling, and sampling, now use the service principal for authentication, ensuring secure and consistent access.

Cloud Vendor Cost Retrieval via API Method

This section explains how to integrate Azure Databricks with the ADOC platform and collect cloud vendor costs via the API method.

Prerequisites

Before you begin, make sure you have:

• An Azure account with the required permissions.
• Access the ADOC platform.
• Azure Databricks workspace created within a resource group.

1. Creating a Service Principal: Refer to the 1. Creating a Service Principal section above.

2. Creating a Custom Role and Assign It to the Service Principal Refer to the 2. Creating a Custom Role and Assign It to the Service Principal above.

• ** Access Control (IAM) in Azure :Ensure that the service principal has the necessary permissions to access cost management data by assigning the custom role in your subscription's Access Control (IAM) settings.
• *
• ** Assign the Custom Role to the Service Principal: Refer to the section Assign custom role to the Service Principal above.
• *

3. Configuring the ADOC Platform: After the Service Principal is set up, configure the ADOC platform to use the API method for cost retrieval.

Onboarding a New Datasource or Updating an Existing One:

1. Open the ADOC platform and select Register from the left pane.

2. Add or Edit a Datasource

   • For a new Datasource: Click Add Data Source.
   • To update an existing Datasource: Click the three dots in the top right corner of the Datasource and select Edit Configuration.

3. Configure the Cost Retrieval Method: Provide the following information:

4. Complete the Setup: Click Submit to save the configuration.

After completing these steps, your ADOC platform will be configured to retrieve cloud vendor costs via the Azure API method. This integration allows for more accurate cost tracking and management within the ADOC environment.

Known Limitations