Databricks (via Direct Connect method)

This page will guide you through the setup of Databricks in K using the direct connect method.

Integration details

Step 1) Databricks access

1. Ensure you have configured Unity enabled catalogue for your workspace

   1. https://community.databricks.com/t5/bangalore/how-do-we-enable-unity-catalog-for-our-workspace/td-p/73258

2. Enable System Schemas for access & queries

   1. Follow the following documentation to enable the system schemas:

      1. https://docs.databricks.com/en/admin/system-tables/index.html#enable

         1. https://kb.databricks.com/unity-catalog/find-your-metastore-id

         2. Run the following commands to enable:

            curl -v -X PUT -H "Authorization: Bearer <PAT TOKEN>" "https://<YOUR WORKSPACE>.cloud.databricks.com/api/2.0/unity-catalog/metastores/<METASTORE ID>/systemschemas/access"
            curl -v -X PUT -H "Authorization: Bearer <PAT TOKEN>" "https://<YOUR WORKSPACE>.cloud.databricks.com/api/2.0/unity-catalog/metastores/<METASTORE ID>/systemschemas/query"
            

3. SQL Warehouse

   1. Create or use an existing SQL Warehouse for K to use for the extraction.

   2. Go to the SQL Warehouse page. Select the SQL Warehouse to use.

      1. Go to the connection details

      2. Note down the Server hostname and HTTP path

4. PAT Token

   1. For the user that will be used to connect, create a PAT token to be used for authentication.

   2. https://docs.databricks.com/en/dev-tools/auth/pat.html

From the above, record down the following for use in the setup:

1. Databricks account URL

   1. e.g. adb-<workspaceId>.<instance>.azuredatabricks.net

2. PAT token

3. Server host name

4. HTTP path

Step 2) Connecting K to Databricks

• Select Platform Settings in the side bar

• In the pop-out side panel, under Integrations click on Sources

• Click Add Source and select Databricks

• Select Direct Connect and add your Databricks details and click Next

• Fill in the Source Settings and click Next

  • Name: The name you wish to give your Databricks source in K

  • Host: Add your Databricks Account location. Can be seen in the URL when you log into your Databricks account.

    • e.g. adb-<workspaceId>.<instance>.azuredatabricks.net

  • Server Hostname: The SQL Warehouse Server hostname

    • e.g. adb-<workspaceId>.<instance>.azuredatabricks.net

  • HTTP Path: The SQL Warehouse Http path either to a DBSQL endpoint or to a DBR interactive cluster

    • e.g. /sql/1.0/warehouses/<warehouseId>

• Add the Connection details and click Save & Next when connection is successful

  • PAT Token: Add the PAT token created in Step 1

• Test your connection and click Save

Step 3) Manually run an ad hoc load to test the integration

• Select Platform Settings in the side bar

• In the pop-out side panel, under Platform Settings click on Sources

• Next to the Source click on the menu icon and select Run

  

• After the manual run is triggered, a pop up bar will appear containing a link that can take you to the Monitor tab where you can check the progress of the job.