---
language: "en"
---
# K Knowledge Base

## Raise a Ticket

*

  ### [Need a new feature?](https://docs.kada.ai/k-knowledge-base/need-a-new-feature.md)

  Is there something you need K to solve for that it currently doesn't do? or an existing feature needs to be enhanced? Raise a feature request to let us know. L...
*

  ### [Report a problem with K](https://docs.kada.ai/k-knowledge-base/report-a-problem-with-k.md)

  Is the K platform down? is it very slow? or changes not being saved? Raise a support ticket with us to let us know. Log in to the KADA Support Portal to raise ...
*

  ### [Can't find what you're looking for?](https://docs.kada.ai/k-knowledge-base/cant-find-what-youre-looking-for.md)

  If you can't find what you need in the Knowledge Library, you can raise a ticket to let us know. Log in to the KADA Support Portal and provide us with the deta...

## Documentation

### [Getting Started](https://docs.kada.ai/k-knowledge-base/getting-started.md)

### [K Features](https://docs.kada.ai/k-knowledge-base/k-features.md)

### [How-To Guides](https://docs.kada.ai/k-knowledge-base/how-to-guides.md)

### [FAQs](https://docs.kada.ai/k-knowledge-base/faqs.md)

### [Deployment \& Setup](https://docs.kada.ai/k-knowledge-base/deployment-setup.md)

### [Integration Guides](https://docs.kada.ai/k-knowledge-base/integration-guides.md)

### [Admin Guides](https://docs.kada.ai/k-knowledge-base/admin-guides.md)

### [What's New](https://docs.kada.ai/k-knowledge-base/what-s-new.md)

*

  ### [Troubleshooting \& Support](https://docs.kada.ai/k-knowledge-base/troubleshooting-support.md)

*

  ### [KADA Software License Agreement](https://docs.kada.ai/k-knowledge-base/kada-software-license-agreement.md)

---
language: "en"
---
# Adding a Data Source

New in Version 6.1  
Only **Workspace Admins \& Platform Admins**can add, edit or delete a Source

Before anyone can import tables, profile data, or create rules, a **Workspace Admin** needs to connect KDQ Agentic to a data source. Everything downstream --- tables, profiles, rules --- depends on this first step.

*** ** * ** ***

## Supported source types

Read access to data is required to perform a DQ Test.  

| **Source type** |                     **What you'll need**                      | **Authentication method supported** |
|-----------------|---------------------------------------------------------------|-------------------------------------|
| PostgreSQL      | Host, port, database name, username, password                 |                                     |
| SQL Server      | Host, port, database name, username, password                 |                                     |
| Snowflake       | Account, username, and either a key pair or OAuth credentials |                                     |
| Amazon Redshift | Host, port, database name, and credentials                    |                                     |
| Databricks      | Server hostname, HTTP path, and catalog name                  |                                     |

*** ** * ** ***

## Adding a source

* Open the workspace and go to **Sources**.

* Click **Add Source** and choose the source type.

* Enter the connection details for your source.

* Run the connection test before saving --- KDQ Agentic checks it can actually reach the source with the details you've entered, and flags anything wrong before you commit to it.

* Save the source once the test passes.

> ⚠️ The authentication method can't be changed after a source is created. Confirm you're using the right method (e.g. key pair vs OAuth for Snowflake) before saving.

Connection secrets are encrypted and masked --- they're never shown again once saved, including to Workspace Admins. If you need to rotate a credential, edit the source and re-enter it; this re-encrypts and re-tests the connection.

*** ** * ** ***

## Keeping a source in sync

* **Rescan** a source to detect schema drift (new, changed, or removed columns) since it was last imported. Anyone in the workspace can view scan run history; only a Workspace Admin can trigger a rescan.

* **Re-discover** a saved source to refresh the list of tables and databases available to import from.

*** ** * ** ***

## Deleting a source

Before you delete a source, KDQ Agentic shows a **cascade impact** view --- everything that depends on it (imported tables, rules, schedule groups) so you know what you're about to affect before you confirm.

*** ** * ** ***

Change history  
**Version 6.1** · 2026-08-17

* NEW Adding and managing KDQ Agentic data sources (PostgreSQL, Snowflake, SQL Server, Redshift, Databricks).

Last updated: August 16, 2026

---
language: "en"
---
# Adding Service Principal to Workspaces via Powershell

The below PowerShell Script can be used to automate the process of adding the created KADA Service Principal to Workspaces to unlock detailed lineage.

## Requirements

* A PowerBI Admin / Fabric Admin account

* Service Principal's Enterprise Object ID

The Object ID required here is the Enterprise App ID. Navigate to **Microsoft Entra ID** \> **Enterprise applications** , select the application, and copy the **Object ID** from the **Overview** tab.  
![image-20251201-041436.png](https://docs.kada.ai/__attachments/a_31df23fd7316ca39015d2001406cf09fd060e4e13a4d3465fb2bd694461ce350/image-20251201-041436.png?cb=bcc533db2625d443a6577c0d27c088fd)

## PowerShell Script

Replace the identifier with the Enterprise Object ID.

    # Install Power BI module
    Install-Module -Name MicrosoftPowerBIMgmt -Scope CurrentUser -Force
    # Import the module
    Import-Module MicrosoftPowerBIMgmt
     # Login with Fabric Admin/PBI Admin account
    Connect-PowerBIServiceAccount
    # Adding Service Principal to All Workspaces
    Get-PowerBIWorkspace -Scope Organization -All -WarningAction Ignore |
        Where-Object {
            $_.State -eq "Active" -and $_.Type -eq "Workspace"
        } |
        ForEach-Object {
            $Workspace = $_
            $Body = @{
                identifier = "{SERVICE PRINCIPAL OBJECT ID}"
                groupUserAccessRight = "Member"
                principalType = "App"
            } | ConvertTo-Json
            try {
                Invoke-PowerBIRestMethod -Method POST `
                    -Url "admin/groups/$($Workspace.Id)/users" `
                    -Body $Body `
                    -ErrorAction Stop
                Write-Host "✓ Added to: $($Workspace.Name)" -ForegroundColor Green
                Start-Sleep -Milliseconds 500
            } catch {
                $errorMessage = $_.Exception.Message
                if ($_.ErrorDetails.Message) {
                    try {
                        $errorObj = $_.ErrorDetails.Message | ConvertFrom-Json
                        $errorMessage = $errorObj.error.message
                    } catch {}
                }
                if ($errorMessage -like "*already*") {
                    Write-Host "○ Already member: $($Workspace.Name)" -ForegroundColor Yellow
                } else {
                    Write-Host "✗ Error on $($Workspace.Name): $errorMessage" -ForegroundColor Red
                }
            }
        }

The above method allows for adding users/applications to workspaces whether or not the Admin account is a member of the workspace.

The below Endpoint will remove the specified Service Principal from specified Workspace if required.

    Invoke-PowerBIRestMethod -Method DELETE `
        -Url "https://api.powerbi.com/v1.0/myorg/groups/{WORKSPACE ID}/users/{SERVICE PRINCIPAL OBJECT ID}"

Last updated: March 13, 2026

---
language: "en"
---
# Adding the Embedded Governance Link into a Power BI report

Updated in Version 6.0

You can add the embedded widget in PowerBI using PowerBI Pro Online, Fabric PowerBI, or PowerBI Desktop.

## Getting the embedded widget link from K

The embedded widget link is available from the Report (or Sheet) profile page in K. The report must be published FIRST and then profiled in K. This process may take up to 24 hours after the publish (as K profiling may take some time to complete).

* Go to the profile page of a report or sheet you want to generate an embedding for.

* In the header, click on the **Menu** icon**.** Select **Generate embedded profile**

  ![image-20260317-131058.png](https://docs.kada.ai/__attachments/a_e2e2b61d6290174838f0eff88ffc29aeb23f594d8dbcd09123568efcae8a4f75/image-20260317-131058.png?cb=39f9a0713e09754c976609c99cf16bab)

* Go to the **HTML CODE.** Set to **Use single quotes**.

* Click **COPY**

![image-20240724-235210.png](https://docs.kada.ai/__attachments/a_2aa6672cf9d70963119475ee0a263139365558a587f585cd09e59e05864411cf/image-20240724-235210.png?cb=3eb3f5567b024b142f687fd502867db0)

### Adding embedded widget link to a Power BI report

In Power BI, we will use the iframe.

* Open the Power BI

* Create a new PowerBI measure by right-clicking on your **Data** , **More options** , and then **New measure**

![image-20241021-055235.png](https://docs.kada.ai/__attachments/a_3dc40acd8abda6bcbe8cbae69bd06c1d3347323848fec4eb873754a7a69e28b8/image-20241021-055235.png?cb=8090eb44beade503fe3d959b7c36b592)

* In PowerBI, paste the code from Step 1 into your measure calculation window.

  ![image-20241021-061318.png](https://docs.kada.ai/__attachments/a_70f95cd1fbe4c25b78a0aa6b002624b888feb4e60bc8de0051cb8e4ffce70daa/image-20241021-061318.png?cb=05a9e35dc3879791449821950d893f4e)
* To display the embedded link you will need to install an HTML component from the PowerBI Visuals Store.

  In the visualisation section of the toolbar select **Get more visuals**.

  ![image-20241021-060407.png](https://docs.kada.ai/__attachments/a_a5aae5c90f08a32c7ef5e6f957a12f07b12d5d57a509e613f573dcd95b410865/image-20241021-060407.png?cb=7c3dc897509991fd10c807ce4580a187)
* Search for [**HTML content**](https://appsource.microsoft.com/en-us/product/power-bi-visuals/WA200001930?tab=Overview) and install it.

  ![image-20241021-060428.png](https://docs.kada.ai/__attachments/a_5be939e8868d9008689f54bea0049f7852248c4cc3843f5fff90f42862fd0537/image-20241021-060428.png?cb=5c8661c663db1a3606d2a07f3ba01c22)
* Click the HTML Content widget to add it onto the canvas. Adjust the HTML content widget position and size on the canvas as required.

  ![image-20241021-060341.png](https://docs.kada.ai/__attachments/a_52936deef831af8f2aa052bc7adb39d4d178c89ea739e8bfa3646722194e2769/image-20241021-060341.png?cb=2d2b6b2b070fe48ec9cb557010e60ae0)
* To display the link, drag the measure into the values section of the HTML content. This should now display your K embedded governance details for this report/sheet.

  ![image-20241021-061437.png](https://docs.kada.ai/__attachments/a_7574d7a1665a83198d423fc4b6bbfa2eba2b25dec7a525c84ca94c881ef745b4/image-20241021-061437.png?cb=02c0713ab794a1bf8480180f6d081718)
* Congratulations, you've now added embedded governance into your report/sheet.

  ![image-20241021-061643.png](https://docs.kada.ai/__attachments/a_dc4795535c6a87b6c84190f87bdb1a5f07038c7b62ea00a4dc509caa5fc990e4/image-20241021-061643.png?cb=9510e494f1d3d379d3729eb9352c6283)

Last updated: July 26, 2026

---
language: "en"
---
# Admin Guides

Updated in Version 6.0

## Platform Administration for KADA Admins

This section is the reference guide for KADA platform administrators --- covering platform settings, user management, data loads, email configuration, and more.

This section is for:

* KADA platform administrators managing the day-to-day operation of the platform

* Admins configuring platform behaviour and system settings

* Anyone responsible for maintaining the health and configuration of your KADA instance

**Note:** This section covers platform administration tasks. For initial deployment and infrastructure setup, see [Deployment \& Setup](https://docs.kada.ai/k-knowledge-base/deployment-setup.md). For connecting data sources, see [Integration Guides](https://docs.kada.ai/k-knowledge-base/integration-guides.md).

### Platform Configuration

|                                                      Guide                                                      |                    Description                    |
|-----------------------------------------------------------------------------------------------------------------|---------------------------------------------------|
| [Platform Settings](https://docs.kada.ai/k-knowledge-base/platform-settings.md)                                                     | Configure global platform settings in KADA        |
| [Setting up email](https://docs.kada.ai/k-knowledge-base/setting-up-email.md)                                                       | Configure email notifications and alerts          |
| [Batch Jobs](https://docs.kada.ai/k-knowledge-base/batch-jobs.md)                                                                   | Manage and monitor scheduled batch jobs           |
| [How to change Search results displayed](https://docs.kada.ai/k-knowledge-base/how-to-change-number-of-search-results-displayed.md) | Change number of displayed results on search page |
| [Connecting a BI Tool](https://docs.kada.ai/k-knowledge-base/how-to-connect-a-bi-tool-to-k.md)                                      | How to configure and connect a BI tool into K     |

### Data Management

|                                                           Guide                                                           |                         Description                         |
|---------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------|
| [How to manually run a data load](https://docs.kada.ai/k-knowledge-base/how-to-manually-run-a-data-load.md)                                   | Trigger a data load outside of the scheduled run            |
| [How to create a collection instance linking rule](https://docs.kada.ai/k-knowledge-base/how-to-create-a-collection-instance-linking-rule.md) | Define rules for automatically linking collection instances |

### User \& Access Management

|                                 Guide                                 |                 Description                  |
|-----------------------------------------------------------------------|----------------------------------------------|
| [Configuring Users in K](https://docs.kada.ai/k-knowledge-base/configuring-users-in-k.md) | Manage user accounts, roles, and permissions |

🆕 **Looking for release notes and version history?** See [What's New](https://docs.kada.ai/k-knowledge-base/what-s-new.md).  
💬 **Need support?** Visit [Troubleshooting \& Support](https://docs.kada.ai/k-knowledge-base/troubleshooting-support.md) to raise a ticket with our team.

Last updated: July 26, 2026

---
language: "en"
---
# Administrator Queries

Updated in Version 6.0

This page provides a list of administrator queries for solving unique problems that are yet to be available via the UI.

*** ** * ** ***

## Dataset upstream source extract

There may be a time when you need to see all the upstream objects for datasets in K.

A current workaround is to follow the below instructions to generate an extract

### Extract details

|        **Columns**        |                 **Description**                 |             **Example**              |
|---------------------------|-------------------------------------------------|--------------------------------------|
| name                      | Name of the dataset                             | Customer model                       |
| object_type               | Type of the dataset                             | Dataset                              |
| object_id                 | ID of the dataset                               | 753f5c32-5cf3-3af6-a499-364d749344e5 |
| source_name               | Name of the dataset source                      | Power BI                             |
| upstream_name             | Name of the upstream object                     | dim_customer                         |
| upstream_object_type      | Type of the upstream object                     | Table                                |
| upstream_object_signature | Fully qualified location fo the upstream object | source.database.schema.table         |
| upstream_id               | ID of the upstream object                       | 8a05f40a-0e6a-3ab1-93eb-9a6db10e0601 |
| upstream_source_name      | Name of the upstream source                     | Snowflake                            |

### Instructions to produce the extract

    # CONNECT to the postgres pod and start a psql session
    kubectl exec -it postgres-statefulset-0  -- psql -U postgres -d cerebrum
    # Run this query
    COPY (
    SELECT DISTINCT
       ds.name AS name,
       dsr.name AS object_type,
       ds.id AS object_id,
       dss.name AS source_name,
       r.name AS upstream_name,
       rr.name AS upstream_object_type,
       r.signature AS upstream_object_signature,
       r.id AS upstream_id,
       rs.name AS upstream_source_name
    FROM node ds
    INNER JOIN node_ref dsr ON dsr.id = ds.node_ref_id
    INNER JOIN source dss ON dss.id =  ds.source_id
    INNER JOIN edge ON edge.source_node_id = ds.id AND edge.source_node_ref_id = ds.node_ref_id
    INNER JOIN node r ON r.id = edge.target_node_id AND r.node_ref_id = edge.target_node_ref_id
    INNER JOIN source rs ON rs.id = r.source_id
    INNER JOIN node_ref rr ON rr.id = r.node_ref_id
    WHERE ds.node_ref_id IN (15,27)
    AND edge.edge_ref_id IN (6,26,32)
    AND edge.source_node_ref_id IN (15,27)
    AND edge.target_node_ref_id IN (3,4)
    AND r.node_ref_id IN (3,4)
    ORDER BY source_name, object_type, name, upstream_source_name, upstream_object_type, upstream_object_signature
    ) TO '/tmp/extract.csv' DELIMITER ',' CSV HEADER;
    # Exist out of postgres pod.
    # Use kubectl to copy the csv out of the pod.
    kubectl cp postgres-statefulset-0:/tmp/extract.csv extract_table_size.csv

Last updated: July 26, 2026

---
language: "en"
---
# Alert Storage Configuration

New in Version 6.1  
This page is for KADA Admins. It covers setting up the storage connections that alerts can export to. To create the alert itself, see **Choosing a delivery channel** on the [Alerts](https://docs.kada.ai/k-knowledge-base/alerts.md) page.

Alerts can deliver by email, or export their results as a CSV to cloud storage. Before anyone can use storage delivery, a KADA Admin needs to set up at least one storage connection here. Once a connection exists, users with the Data Governance, Data Manager, or KADA Admin role can select it when creating an alert.

K currently supports two storage types: Amazon S3 and Azure Blob.

*** ** * ** ***

## Creating a storage connection

* On the side menu, click **Settings** and under Integrations, click **Storage**.

  * The page lists every connection that's been configured, with its name and type (S3 or Azure Blob), sorted alphabetically by name.

![image-20260810-121735.png](https://docs.kada.ai/__attachments/a_1424d6dc7c4e0dd95c0ddfe4cd7225e3a0774b0b2d3aaa7c9a522cfdb1f9fbfe/image-20260810-121735.png?cb=d794c216aed689b3c7504db114cffe18)

* Click **Add connection** and a pop-up will appear to create a new storage connection

![image-20260810-121952.png](https://docs.kada.ai/__attachments/a_881201d924b8ae0f9b3c8f8345ec7e319133175883f7c405a4fb67ff518a3e02/image-20260810-121952.png?cb=03b884f26a6cad67a2d4d28fc9f2a980)

* Enter a **Name** for the connection. This is what shows up in the delivery dropdown when someone creates an alert.

* Select a **Template**: S3 or Azure Blob.

* Select an **Authentication type** for that template. Amazon S3 currently supports Access Key.

* Fill in the fields for your template and authentication type:

  * **S3 with Access Key:** bucket name, region, access key ID, secret access key

  * **Azure Blob:** container name, account name, access key

* Click **Test connection**. K checks that it can both write to and read from the destination.

* Click **Save**.

**Save stays disabled** until a test has passed against your current field values. If you change any credential field after testing, you'll need to test again before you can save.

*** ** * ** ***

## Viewing and editing a connection

Click anywhere on a connection's row to see its details.

Click **Edit** from that view to change the name or configuration. As with creating a connection, you'll need to re-test before Save is enabled if you change any credentials.  
Note: Credential fields are always masked here, they never show the actual value.

*** ** * ** ***

## Deleting a connection

Open the actions menu on a connection and click **Delete**.

* If any alerts are still using that connection, deletion is blocked. K tells you how many alerts are attached, clean those up (or point them elsewhere) first.

* If nothing is using it, you'll be asked to confirm before it's removed.

*** ** * ** ***

Change history  
**Version 6.1** · 2026-08-17

* New Storage admin page for KADA Admins to configure reusable S3 and Azure Blob connections that alerts can export to.

Last updated: August 10, 2026

---
language: "en"
---
# Alerts

UPDATED IN VERSION 6.1

K can send alerts triggered from any saved search. They are a great way to automate reminders and prompts. For example:

* Sharing K Alert outputs with non-K users

* Scheduling reports to trigger BAU operational processes (Weekly scheduled extract of all data assets with open governance issues to trigger follow-up activity)

This page covers:

* Alert types: Custom and Platform

* Creating a Custom alert

* Choosing a delivery channel

* Setting the schedule

* Managing your alerts

*** ** * ** ***

## Alert Types

The Alerts page shows two types of alert:  

|            Type             |                             What it's for                              |           Where it's created           |
|-----------------------------|------------------------------------------------------------------------|----------------------------------------|
| Saved Search (Custom alert) | Triggered from any saved search you own. Covered on this page.         | Saved Search page                      |
| DQ Alerts (Platform alert)  | Built-in DQ alerts: "DQ Failed Tests" and "DQ Tests requiring review". | Data Quality Dashboard's My Alerts tab |

*** ** * ** ***

## Creating a Custom K Alert

A K alert can be scheduled through the **Saved Search**page.  
![image-20260810-114159.png](https://docs.kada.ai/__attachments/a_0943781848235840e2189be98cf53c32134d3faaabea7e0fc002d584d363898b/image-20260810-114159.png?cb=d64c58c2be49d4d30a6686982ef3a6dc)

* Open the **more actions** menu and click **Create Alert.**

* A pop-up box will appear asking you to confirm:

|      Field      |                                                            Details                                                             |
|-----------------|--------------------------------------------------------------------------------------------------------------------------------|
| Alert name      | Required, must be unique to you                                                                                                |
| Delivery option | Email or storage, see [**Choosing a delivery channel**](https://docs.kada.ai/k-knowledge-base/alerts.md#Choosing-a-delivery-channel) below         |
| Schedule        | Daily, weekly, monthly, or custom cron, see [**Setting the schedule**](https://docs.kada.ai/k-knowledge-base/alerts.md#Setting-the-schedule) below |
| Run alert now   | Off by default. Turn it on to initiate the alert immediately on save                                                           |

![image-20260810-114503.png](https://docs.kada.ai/__attachments/a_448d42772675b4596c479f125d720e7de35cfe5d8f0c4cb4b5a2ed0f38e5dbc2/image-20260810-114503.png?cb=21a138b9211848f35781f48554f4fd53)

* Click **Create alert** to save it. To change an existing alert later, open it from the Alerts page, it opens the same modal, pre-filled.

*** ** * ** ***

## Choosing a delivery channel

An alert delivers to exactly one channel, email or storage. You can't send the same alert to both.

### Email

* You're added as a recipient by default.

* Add more K users by username, or type any email address.

* The email links back to the exact search results that triggered it.

Email delivery needs SMTP configured on the platform. If it isn't, the email option is disabled here and a banner appears on the Alerts page. See [Setting up email](https://docs.kada.ai/k-knowledge-base/setting-up-email.md).

### Storage

Storage delivery exports the matching results as a CSV, packaged as ZIP or GZ, to a cloud storage location. It's only available to **KADA Admins** , **Data Governance** , and **Data Manager** roles, useful for feeding results into a downstream pipeline.

* Select one pre-configured storage connection (S3 or Azure Blob).

* Enter the path within that connection, e.g. `/exports/alerts/`.

* Choose ZIP or GZ as the file format.

Storage connections are set up separately by a KADA Admin, under **Settings \> Integrations \> Storage** . See [Alert Storage Configuration](https://docs.kada.ai/k-knowledge-base/alert-storage-configuration.md). If none exist yet, this option is disabled.

### Zero-result alerts

If a scheduled run finds no matching results, nothing is sent or written, no empty email, no empty file. The alert's last run status shows as **Completed (Not Sent)**.

*** ** * ** ***

## Setting the schedule

|  Frequency  |                              What you set                              |
|-------------|------------------------------------------------------------------------|
| Daily       | A time of day, or "load time"                                          |
| Weekly      | Day of week + time                                                     |
| Monthly     | Load day, 1st of month, or a specific day-of-week, + time              |
| Custom cron | A cron expression. K shows the resolved next-run time once it's valid. |

### Run now

Turn on **Run now** in the create/edit modal to fire the alert immediately, on top of its normal schedule. You can also trigger this later from the Alerts page by clicking **Run now**.  
![image-20260810-115205.png](https://docs.kada.ai/__attachments/a_cc6445d6fd2890f8414b5f1af3654c0316f2ddbe49cd3a1382c1fb8e353a35de/image-20260810-115205.png?cb=bb440dadab24d07b1d349e893468cfb2)

Either way, K takes you straight to that alert's run history so you can watch it complete.  
You can't run an alert that's already running. K blocks the second run and shows an error until the first one finishes.

*** ** * ** ***

## Managing your alerts

In the left navigation, open **My workspace** and click **Alerts** to see every alert you have access to.  
![image-20260810-120402.png](https://docs.kada.ai/__attachments/a_4116f3b3b47fadabdfc5d34ca0febf4a6cec16202e6b4cca85597ada01bed7af/image-20260810-120402.png?cb=967aa57015d0efc878535b350e6b9bcc)

Use the search box to find an alert by name, or filter by:

* **Last run status** --- all statuses, not run, running, completed (sent), completed (not sent), or failed

* **Created by** --- KADA Admins only: yours, or everyone's

* **Alert type** --- all types, DQ Failed Tests, DQ Tests requiring review, or Saved Search

* **Enabled** --- enabled (default), disabled, or all

You can edit or delete only the alerts you created. KADA Admins see everyone's alerts and can turn any of them on or off, but can't edit or delete an alert someone else created.  
Deleting an alert also permanently deletes its run history. If the saved search behind a Custom alert is deleted, the alert and its history are removed automatically. You can't delete an alert while it's running.

*** ** * ** ***

Change history  
**Version 6.1** · 2026-08-17

* Updated Alerts now support storage delivery

* New The Alerts page now includes Platform (DQ) alerts

* New Each alert has a detail page with full run history, a Refresh action, and per-run delivery detail.

* Updated Alerts that find zero results no longer send an empty email or file, they're recorded as Completed (Not Sent) instead.

Last updated: August 15, 2026

---
language: "en"
---
# API

K provides a rich API for programmatically interacting with the Data Items that make up your Data Ecosystem.

*** ** * ** ***

## API Specification

Our API specification is available on our restricted support portal.

Reach out to us for access or to receive a copy of the K API Specification.

It includes API descriptions for all the available REST APIs including:

* Available endpoints (`/object`) and operations on each endpoint (`GET /object`, `POST /object`)

* Operation parameters --- input and output for each operation

* Authentication methods

*** ** * ** ***

## Getting Started

Check out the API Guides sub-page on how to use the API.

Last updated: March 08, 2026

---
language: "en"
---
# API Guide: Creating and Linking Issues

API calls require an Authorisation header Bearer token.

See [API Guides \| Get Access Token](https://docs.kada.ai/k-knowledge-base/api-guides.md#Get-Access-Token)

## Create an issue

Valid values for workflow_status: `To do, Under investigation, Pending, In progress, Done, Cancelled, Declined`

Valid values for priority: `Highest, High, Medium, Low, Lowest`

Request

    POST https://{your.domain.com}/api/issues
    {"name":"issue name","description":"add a description here","workflow_status":"To do","priority":"Highest"}

Response

    {
        "active_flag": true,
        "alternate_name": "",
        "created_at": "2025-06-12T00:18:17.509413+00:00",
        "description": "add a description here",
        "display_id": 163,
        "external_id": null,
        "external_url": null,
        "first_used_at": null,
        "frequency": 0.0,
        "id": "8022aee8-f5b3-3598-9a28-53162e9c192c",
        "last_used_at": null,
        "location": "",
        "manual_creation": false,
        "merge_type": null,
        "name": "issue name",
        "object": {
            ...
        },
        "parent": null,
        "parent_id": null,
        "priority": "Highest",
        "reference": false,
        "score": 0.0,
        "signature": "4739a79",
        "source_id": 1000,
        "updated_at": "2025-06-12T10:18:17.526017+10:00",
        "workflow_status": "To do"
    }

### Link issues to object

`issue.id` is the id returned from the create issue response.

`object.id` is the object id returned from List of objects

    PUT https://{your.domain.com}/api/issues/<issue.id>/related?object_id=<object.id>&relationship=IMPACTS

Last updated: March 22, 2026

---
language: "en"
---
# API Guide: Search assets by Collection instance & Get Steward Details

API calls require an Authorisation header Bearer token.

See [API Guides \| Get Access Token](https://docs.kada.ai/k-knowledge-base/api-guides.md#Get-Access-Token)

## Section 1: Search assets by Collection instance

Return list of objects

Example: Return Reports and Datasets filtered on category:gold

Paginate using start and rows.

Example rows: 30 means 30 results returned per page.

start: 0 means return 30 rows start from the first row.

To return the next page increment start by rows. eg start: 30

Request

    POST https://{your.domain.com}/api/v2/index/search/select
    {
        "params": {
            "q": "*",
            "fq": "asset:(REPORT DATASET) AND collection_category:(GOLD)",
            "fl": "name,location,id,stewards_ids",
            "defType": "edismax",
            "wt": "json",
            "sort": "name asc",
            "start": 0,
            "rows": 30
        }
    }

Response
JSON

    {
        "responseHeader": {
            "status": 0,
            "QTime": 1
        },
        "response": {
            "numFound": 169,
            "start": 0,
            "numFoundExact": true,
            "docs": [
                {
                    "location": "test",
                    "name": "Report 1234",
                    "id": "1b96c222-a782-3fba-ad06-d9dc31352671",
                    "stewards_ids": [
                        "223b7f4c-a646-32a3-b0b3-d83a26716199",
                        "9f2ab0c1-3d4e-3a5b-8c7d-1e2f3a4b5c6d"
                    ]
                }
            ]
        }
    }

Note: stewards_ids is a list of user ids, one asset can have multiple stewards.\*\*

### Section 2: Get data steward email

Use the same search endpoint, filtering on the steward ids from Section 1. Returns name and email per steward.

The following POST will return the details for a single steward

    POST https://{your.domain.com}/api/v2/index/search/select
    {
        "params": {
            "q": "*",
            "fq": "id:223b7f4c-a646-32a3-b0b3-d83a26716199",
            "fl": "id,name,email",
            "wt": "json"
        }
    }

The following example will return the details for multiple stewards

    Get multiple stewards

    ```
    POST https://{your.domain.com}/api/v2/index/search/select
    {
        "params": {
            "q": "*",
            "fq": "id:(223b7f4c-a646-32a3-b0b3-d83a26716199 9f2ab0c1-3d4e-3a5b-8c7d-1e2f3a4b5c6d)",
            "fl": "id,name,email",
            "wt": "json"
        }
    }

Response

    {
        "responseHeader": {
            "status": 0,
            "QTime": 1
        },
        "response": {
            "numFound": 2,
            "start": 0,
            "numFoundExact": true,
            "docs": [
                {
                    "id": "223b7f4c-a646-32a3-b0b3-d83a26716199",
                    "name": "Test User One",
                    "email": "TESTUSER1@EXAMPLE.COM"
                },
                {
                    "id": "9f2ab0c1-3d4e-3a5b-8c7d-1e2f3a4b5c6d",
                    "name": "Test User Two",
                    "email": "TESTUSER2@EXAMPLE.COM"
                }
            ]
        }
    }

Last updated: July 23, 2026

---
language: "en"
---
# API Guides

## Get Access Token

The access token is used in the request header `Authorization: Bearer <COPY ACCESS CODE HERE>`

    # REPLACE: <USER>, <PASSWORD>
    curl -X POST -u "know-app:" \
      -d "grant_type=password&username=<USER>&password=<PASSWORD>&scope=openid profile email" \
      https://<domain>/keycloak/auth/realms/kada/protocol/openid-connect/token
      
    # Response. The access_token will be used to authenticate API calls to KADA.
    {
        "access_token": "..",
        "expires_in": 7200,
        "refresh_expires_in": 86400,
        "refresh_token": "...",
        "token_type": "bearer",
        "id_token": "...",
        "not-before-policy": 0,
        "session_state": "...",
        "scope": "openid profile email"
    }

    export ACCESS_TOKEN=<ACCESS TOKEN FROM RESPONSE>

## Get an object

Get the table with name 'customer'

    curl --location \
      --request GET 'https://<domain>/api/tables?name=customer' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}"

Get fully qualified table name: table123, in schema3 in db2 in host1.

    curl --location \
      --request GET 'https://<domain>/api/tables?signature=host1.db2.schema3.table123' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}"

## Updating a description

`<TABLE ID>` can be found from the Get object call.

    curl --location --request PUT 'https://<domain>/api/tables/<TABLE ID>' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}" \
      --header 'Content-Type: application/json' \
      --data-raw '{
        "description": "Add your description here",
        "replace_date": null
    }'

## Adding a tag to a table

Create a tag

    curl --location --request POST 'https://<domain>/api/tags' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}" \
      --header 'Content-Type: application/json' \
      --data-raw '{
        "description": "Add your tag description here",
        "name": "tag_name"
    }'

Link the tag to an object.

`<TABLE ID>` is returned from the Get object call

`<TAG ID>` is returned from the Get Tags or Create tag call

    curl --location \
      --request PUT 'https://<domain>/api/tables/<TABLE_ID>/related?object_id=<TAG_ID>&relationship=TAGGED_BY' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}"

Delete a tag from an object

    curl --location \
      --request DELETE 'https://<domain>/api/tables/<TABLE_ID>/related?object_id=<TAG_ID>&relationship=TAGGED_BY' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}"

## Adding a property to an object

A property name is unique for a given object.

`<OBJECT ID>` can be found from the Get object APIs.

    curl --location \
      --request PUT 'https://<domain>/api/additionalproperties/<OBJECT ID>' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}" \
      --header 'Content-Type: application/json' \
      --data-raw '[
        {
            "operation": "ADD_OR_UPDATE",
            "name": "new property 123",
            "type": "STRING",
            "value": "PASS",
            "description": "Add a property description here"
        },
        {
            "operation": "CLEAR",
            "name": "existing property",
            "type": "STRING",
            "value": "",
            "description": ""
        }
    ]'

## Get a collection template

Get the Classification collection.

    curl --location \
      --request GET 'https://domain/api/collections?name=Classification' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}"

Response - note the properties which will be used to create collection instances.
JSON

    {
        "id": "e1fd2337-1b0e-3841-8d60-7c85daa1707e",
        "id_seq": 2,
        "properties": [
            {
                "allowed_values": [],
                "data_type": "TEXT_FIELD",
                "description": "Add a name for the instance",
                "id": "1",
                "name": "name",
                "required": true
            },
            {
                "allowed_values": [],
                "data_type": "TEXT_BOX",
                "description": "Add a description for the instance",
                "id": "2",
                "name": "description",
                "required": false
            },
            {
                "allowed_values": [],
                "data_type": "USER_LOOKUP_FILTERED",
                "description": "Add an owner for the instance",
                "filter": "16e05af2-13fa-301d-b7fa-69d48bc71d7d",
                "id": "-1",
                "name": "data owner",
                "required": false
            },
            {
                "allowed_values": [],
                "data_type": "USER_LOOKUP_FILTERED_MULTI",
                "description": "Add steward(s) for the instance",
                "filter": "d13d6b10-a535-3718-854d-459f086ad057",
                "id": "-2",
                "name": "data steward",
                "required": false
            }
        ],
        "score": 5.0,
        "short_name": "classification",
        "signature": "platform/classification",
        "source_id": 1000,
        "updated_at": "2021-10-14T16:02:13.139799+00:00"
    }

## Creating a collection instance

First create a collection and define the template in KADA via the Admin portal.

Use the `properties` field from the Get collection template to map the properties values to the property id.

`<COLLECTION ID>` Use Get collection template to find the collection id.

First create the collection instance:

    curl --location --request POST 'https://<domain>/api/collectioninstances' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}" \
      --header 'Content-Type: application/json' \
      --data-raw '{
      "collection_id": "<COLLECTION ID>",
      "description": "Open information",
      "name": "Public",
      "properties": {
        "1": "Public",
        "2": "Open information",
        "-1": "",
        "-2": []
      }
    }'

## Linking the collection instance to a table

`<TABLE ID>` the table being linked to the collection

`<COLLECTION INSTANCE ID>` the collection instance being linked.

    curl --location \
      --request PUT 'https://<domain>/api/tables/<TABLE ID>/related?object_id=<COLLECTION INSTANCE ID>&relationship=MEMBER_OF' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}"

## Adding a Steward to a table

Similarly to add an owner use the relationship `relationship=OWNED_BY`

    curl --location \
      --request PUT 'https://<domain>/api/tables/<TABLE ID>/related?object_id=<user-id>&relationship=STEWARDED_BY' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}"

## Creating manual lineage to an upstream table

`<TABLE ID>` this is the downstream table. Use Get to find the table id.

`<UPSTREAM TABLE ID>` this is the source table. Use Get to find the table id.

    curl --location \
      --request PUT 'https://<domain>/api/tables/<TABLE ID>/related?object_id=<UPSTREAM TABLE ID>&relationship=SOURCE_FROM' \
      --header "Authorization: Bearer ${ACCESS_TOKEN}"

Last updated: March 08, 2026

---
language: "en"
---
# Ask K

UPDATED IN VERSION 6.1

Ask K is a self-service feature that answers questions about your data ecosystem.

Instead of querying databases or compiling reports manually, data managers, governance users, and administrators can select from pre-built question categories to get instant answers --- and schedule those answers to be delivered automatically.

*** ** * ** ***

## How to use Ask K

**Step 1)** Ask K is a targeted export query. Using Search and the filters settings, narrow down the search results to the data assets you want to perform the Ask K query on.  
![image-20260312-105143.png](https://docs.kada.ai/__attachments/a_0fd183c5647144701730567d7897dea5ea63c207a4f4a349b521b7f40a01dd0f/image-20260312-105143.png?cb=d9276b4fa90c27a990d134a9985c77df)

**Step 2)** Refine your Ask K export and then click **Generate Export**

* Choose your preferred template

* Customise the export configuration

* In the excel export, you can choose to add additional columns, or for the non-mandatory columns, click on the 'x' to remove the column

![image-20260312-105922.png](https://docs.kada.ai/__attachments/a_08869ef79be65e2d5230fe755c93f8c60fabc33c3215c46702ff37e1ebb83931/image-20260312-105922.png?cb=69cd24ce474f64b91d2d15f285c8bc67)

**Step 3)** Click **Download**to save your report. If your report takes a while to load, an automated email will be sent to you when your export is ready to download.  
![image-20260312-105513.png](https://docs.kada.ai/__attachments/a_69e8571ff0e7b0ddc9617c17b2fcf0f6e3d6a63f614f02d0e8b520038ce42c92/image-20260312-105513.png?cb=43f4db7fd15a0b74fd7bfac90e5ee56c)

*** ** * ** ***

## About the different Export Templates

Ask K has 7 question categories, each covering a different area of your data ecosystem:  

|               **Category**               |     **Export Template**      |                                                                                             **What you can answer**                                                                                              |          **Applicable to**          |
|------------------------------------------|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|
| **Properties export**                    | Asset details                | A quick way to easily extract a list of properties about the data assets. The default list of properties include: * ID * NAME * OBJECT TYPE * LOCATION * OBJECT URL Add any additional properties to the export. | All assets                          |
|                                          | Governance details           | A template with key governance properties included e.g. Owner, Classification, Domain, and Verified User Cases                                                                                                   | Data, content and collection assets |
| **Usage exports**                        | Asset usage summary          | Key usage stats for each asset include number of usage read, writes, modify, run, and download over a specified period                                                                                           | Data and content assets             |
|                                          | Asset usage by user          | A list of users that has read, write, modified, run or downlaoded asset during a specified period Applicable to data and content assets                                                                          | Data and content assets             |
|                                          | User usage by asset          | A list of data assets for each user has read, write, modified, run or download during a specified period                                                                                                         | Users                               |
|                                          | Asset usage by tool          | A list of tools that reads, writes, modifies, runs or downloads for each asset Applicable to data and content assets                                                                                             | Data and content assets             |
| **Transformation exports**               | Upstream load logic          | Export code responsible for loading data assets i.e. how the data gets created or populated. Use this when you need to trace where the asset's data originates from upstream sources                             | Data and content assets             |
|                                          | Downstream consumption logic | Export code that reads or consumes data assets. Use this when generating semantic models as it shows how the asset is actually used downstream                                                                   | Data and content assets             |
| **Access exports**                       | Asset access by user         | A list of users that has access to, and their usage of, each asset Applicable to data and content assets                                                                                                         | Data and content assets             |
|                                          | User access by asset         | A list of assets that each user has access to and their usage details for each asset Applicable to users                                                                                                         | Users                               |
| **Sensitive data scanner configuration** | K data scanner scope         | List of tables that the PII Sensitive data scanner is configured to monitor                                                                                                                                      | Tables only                         |

*** ** * ** ***

Change history  
**Version 6.1** · 2026-08-17

* New Added two Transformation export templates: Upstream load logic and Downstream consumption logic

Last updated: August 11, 2026

---
language: "en"
---
# Assets Catalogued in K

Updated in Version 6.0

K catalogues assets from across your data ecosystem, classifying them into two primary types: **Data** and **Content**.

* **Data** assets represent structured data stored in databases and schemas (e.g. tables, columns).

* **Content** assets represent analytical and reporting artefacts built on top of data (e.g. reports, dashboards, pipelines, ML models).

Various objects within each integrated source are automatically allocated to an object type. For example:

* A Power BI workspace is catalogued as a *Workspace*

* A Power BI app is catalogued as a *Content App*

* A Power BI data flow is catalogued as a *Pipeline*

* A Redshift External table is catalogued as a *Table* (Table type: External)

* A Snowflake View is catalogued as a *Table* (Table type: View)

*** ** * ** ***

## Data Assets

|                                                                                     **Object Type**                                                                                     |                                      **Description**                                       |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
| Database ![image-20260207-121456.png](https://docs.kada.ai/__attachments/a_e0c3d0f536e7b8ca12c03cd596f8888a195b88500fb37be6cf2451222968b2f8/image-20260207-121456.png?cb=cc9494cccb131cc7ee9e5d1b6f88f096)  | The database used to store and manage structured data assets                               |
| Schema ![image-20260207-121525.png](https://docs.kada.ai/__attachments/a_b12c748f5071d3f19d85640ba2836da6f37171b8336139d2f85f38e969f60f07/image-20260207-121525.png?cb=6616347ccfdf4d895da741e24121c0f3)    | A logical grouping within a database that organizes related tables, views, and procedures. |
| Table ![image-20260207-121549.png](https://docs.kada.ai/__attachments/a_24f4d699ae0a182bf6aceae26d469ad282929a8822cff45d6d433905d5a9d532/image-20260207-121549.png?cb=f4b6926c9de5fe92fb37f5e3ee9548d4)     | A data object within a schema that stores records in rows and columns.                     |
| Column ![image-20260207-121621.png](https://docs.kada.ai/__attachments/a_b0fcb81d473adfb4e1d0924b079ba6c49b8991c0db4f6450211968de46bb0c9f/image-20260207-121621.png?cb=9e44e17180a6be5a166421418bd2b189)    | A single field or attribute within a table that defines a specific data element.           |
| Code ![image-20260207-120601.png](https://docs.kada.ai/__attachments/a_1599feb73143c517449aaaa661e5a471d547d78282983c2aee9f357606a04cf6/image-20260207-120601.png?cb=3c38ecb4905c79964f66280406e27f72)      | A script or program that performs data processing, transformation, or analysis.            |
| Procedure ![image-20260207-120601.png](https://docs.kada.ai/__attachments/a_1599feb73143c517449aaaa661e5a471d547d78282983c2aee9f357606a04cf6/image-20260207-120601.png?cb=3c38ecb4905c79964f66280406e27f72) | Stored routine that encapsulates a series of SQL statements or operations.                 |
| Macro ![image-20260207-120601.png](https://docs.kada.ai/__attachments/a_1599feb73143c517449aaaa661e5a471d547d78282983c2aee9f357606a04cf6/image-20260207-120601.png?cb=3c38ecb4905c79964f66280406e27f72)     | Parameterized or reusable block of logic that simplifies repetitive operations.            |

*** ** * ** ***

## Analytical \& Reporting Content Assets

|                                                                                       **Object Type**                                                                                       |                                                                   **Description**                                                                    |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
| Tool ![image-20260207-120835.png](https://docs.kada.ai/__attachments/a_7abb07c6fa795e2ce9face1f38a92c51bdaf016e456c4eadeada343124efb8b3/image-20260207-120835.png?cb=120e6e6179808d824ccccf638a4fc663)          | The application or platform used to create, process, or manage analytical \& reporting content assets within the data ecosystem                      |
| Workspace ![image-20260207-122107.png](https://docs.kada.ai/__attachments/a_5f91fcadeaab179bbdc341328d21da8a66906949da823a352d979a1413a9b498/image-20260207-122107.png?cb=4e2ee406c3f86bad3f29eac2e293dfc2)     | A logical container that groups related assets such as datasets, reports, and pipelines.                                                             |
| Report ![image-20260207-121116.png](https://docs.kada.ai/__attachments/a_5ab5c7c9eff8325e7b1c38b3841b6fc09a3522817800c39c8dadd34ae2749842/image-20260207-121116.png?cb=9474e09851c93b69c30bf965d73b7be7)        | A dashboard, report or document that presents data, charts or insights. Typically connected to one or more datasets.                                 |
| Sheet ![image-20260207-121145.png](https://docs.kada.ai/__attachments/a_2c3f36387f778bf042dce5fac7271cf956ad17fbcd2c8eae30781ab72932d760/image-20260207-121145.png?cb=0457a560bd1e66cf5895c1218f66e65a)         | A page within a report                                                                                                                               |
| Content App ![image-20260207-121237.png](https://docs.kada.ai/__attachments/a_9a1317d0c181e3e86b06d389284b51771a60503ddef59f6b3f9cec3bf2ae3745/image-20260207-121237.png?cb=01f770cfa828f838f922541c0e77de84)   | A logical container that groups assets used for data consumption (e.g. Reports).                                                                     |
| Pipeline ![image-20260207-121728.png](https://docs.kada.ai/__attachments/a_0c690a2f58280aeb02ba7caf4c706bcf4012009d3dbd99141db0acf5ea9e54e7/image-20260207-121728.png?cb=c8a62d6354d00a00a228454ecb9b606f)      | A data processing flow that moves, transforms, or integrates data between datasets.                                                                  |
| ML Model ![image-20260207-121036.png](https://docs.kada.ai/__attachments/a_25cbd5cee333cff2f724f86dbdd89cd4d2a471cad4acf245070cfe4a0a0e78f1/image-20260207-121036.png?cb=2935e4be91a7cecc3149129dbda7a624)      | A machine learning artifact that uses input data to make predictions or classifications based on historical patterns.                                |
| Dataset ![image-20260207-121317.png](https://docs.kada.ai/__attachments/a_db16a9d98c03708adea2404fdafd3d8c8d22f6e89a0af3f910d282502a40465b/image-20260207-121317.png?cb=ec23a32bcc55dfe0112f7248939a4aa8)       | A structured collection of data used for reporting, analysis, or modeling. It typically serves as the foundation for reports or ML models.           |
| Dataset Table ![image-20260207-121342.png](https://docs.kada.ai/__attachments/a_817e298929555ea01286af64098b82aa543af215603ef2c841db25af0a609645/image-20260207-121342.png?cb=0fa810c02a871c38a458001e3dba0474) | A table within a dataset that organizes data into rows and columns, often representing a specific entity or subject area.                            |
| Dataset Field ![image-20260207-121412.png](https://docs.kada.ai/__attachments/a_6429ef1fe7e0651161562b7fcf9c8276c60b07b35604d6962fba69c7b68b54c8/image-20260207-121412.png?cb=2024308ce895a829ab3fda48c6a956a9) | An individual column or attribute within a dataset table that stores a specific data element (e.g., Customer Name, Order Date).                      |
| File ![image-20260207-121807.png](https://docs.kada.ai/__attachments/a_130777a6bf8605ddd01ec846b08366bda181505afe7dbb0e96880426ea144367/image-20260207-121807.png?cb=bdad102a1edaa46d11f616a900ff9a0a)          | An asset stored in a filesystem or object store, often in formats such as CSV, JSON, or Parquet, used as input or output for analysis and processing |

*** ** * ** ***

## Asset and Object Type Hierarchy

| **Asset Type** | **Dataset / Pipeline** | **Object Type** |  **Object Children**  |
|----------------|------------------------|-----------------|-----------------------|
| Data assets    | Schema                 | Table           | Column                |
| Data assets    | Schema                 | Macro           | ---                   |
| Data assets    | Schema                 | Procedure       | ---                   |
| Content assets | Content                | Content Child   | ---                   |
| Content assets | Dataset                | Dataset Table   | Dataset Field         |
| Content assets | Dataset Pipeline       | ---             | ---                   |
| Content assets | Content App            | Content         | Content Child         |
| Content assets | ML Model               | ---             | ---                   |
| Content assets | Workspace              | Content         | Content Child         |
| Content assets | Workspace              | Dataset         | Dataset Table / Field |
| Content assets | DQ Test                | ---             | ---                   |

Last updated: July 26, 2026

---
language: "en"
---
# Assigning an Owner/Steward to a Data Quality Test

Updated in Version 6.0

K lets you assign Owners/Stewards to a Data Quality Test.

The benefit is to provide Owners/Stewards with visibility of the DQ Tests that are relevant for their work. They can also configure alerts for their DQ Tests.

*** ** * ** ***

## How to assign an Owner / Steward to a Data Quality Test

* Open a Data Quality Test profile

* Click on the **Owner** or **Steward** field in the sidebar

* Select an Owner or Steward. More than one Owner or Steward can be assigned

Last updated: July 26, 2026

---
language: "en"
---
# Athena (via Collector method)

This page outlines the Athena Collector versions that are available.

We always recommend using the latest Collector Method to ensure that you can access the latest features.

## Integration details

|       **Scope**        | **Included** | **Comments** |
|------------------------|--------------|--------------|
| Metadata               | YES          |              |
| Lineage                | YES          |              |
| Usage                  | YES          |              |
| Sensitive Data Scanner | N/A          |              |

*** ** * ** ***

## Athena Version History

|                       **Version Number**                        | **Release Month** | **K Platform Compatibility** | **Collector Core Library Compatibility** |                  **Release changes**                   |
|-----------------------------------------------------------------|-------------------|------------------------------|------------------------------------------|--------------------------------------------------------|
| [V3.1](https://docs.kada.ai/k-knowledge-base/athena-via-collector-method-v3-1-0.md) | Jun 2026          | 6.0+                         | 1.2+                                     | Added options for meta only and events only extraction |
| [V3.0](https://docs.kada.ai/k-knowledge-base/athena-via-collector-method-v3-0-0.md) | Nov 2022          | 5.23 - 5.25                  | 1.0                                      | First version released                                 |

Last updated: July 15, 2026

---
language: "en"
---
# Athena (via Collector method) - v3.0.0

## About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

### Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it to pull data for security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pulls the latest metadata

3. Storing and pushing the extracts to your K instance

*** ** * ** ***

## Pre-Requisites

**Collector Server Minimum Requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**Athena Requirements**

* Access to **Athena**

*** ** * ** ***

## Step 1: Establish Athena Access

It is advised you create a new Role and a separate s3 bucket for the service user provided to KADA and have a policy that allows the below, see [Identity and access management in Athena - Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/security-iam-athena.html)

The service user/account/role will require permissions to the following

1. Execute queries against Athena with access to the INFORMATION_SCHEMA in particular the following tables:

   1. information_schema.views

   2. information_schema.tables

   3. information_schema.columns

2. Executing queries in Athena requires an s3 bucket to temporarily store results. We will also require the policy to allow Read Write Listing access to objects within that bucket, conversely, the bucket must also have policy to allow to do the same.

3. Call the following Athena APIs (Note that access to Athena metadata through the below APIs will also require access to the Glue catalog).

   1. [BatchGetQueryExecutions](https://docs.aws.amazon.com/athena/latest/APIReference/API_BatchGetQueryExecution.html)

   2. [GetQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html)

   3. [GetQueryResults](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html)

   4. [ListQueryExecutions](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListQueryExecutions.html)

   5. [StartQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_StartQueryExecution.html)

   6. [ListWorkGroups](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListWorkGroups.html)

   7. [ListDataCatalogs](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListDataCatalogs.html)

   8. [ListDatabases](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListDatabases.html)

   9. [ListTableMetadata](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListTableMetadata.html)

4. The service user/account/role will need permissions to access all workgroups to be able to extract all data, if you omit workgroups, that information will not be extracted and you may not see the complete picture in K.

5. See [IAM policies for accessing workgroups - Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/workgroups-iam-policy.html) on how to add policy entries to have fine grain control at the workgroup level. Note that the extractor runs queries on Athena, If you do choose to restrict workgroup access, ensure that Query based actions (e.g. StartQueryExecution) are allowed for the workgroup the service user/account/role is associated to.

Note that user usage will be associated to the workgroup level rather than individual users, these workgroups are published as users in K in the form "athena_workgroup_\<name\>"

Example Role Policy to allow Athena Access with least privileges for actions, this example allows the **ACCOUNT ARN** to assume the role. Note the variables **ATHENA RESULTS BUCKET NAME.** You may also choose to just assign the policy directly to a new user and use that user without assuming roles. In the scenario you do wish to assume a role, please note down the role ARN to be used when onboarding/extracting.

    AWSTemplateFormatVersion: "2010-09-09"
    Description: 'AWS IAM Role - Athena Access to KADA'
    Resources: 
      KadaAthenaRole: 
        Type: "AWS::IAM::Role"
        Properties: 
          RoleName: "KadaAthenaRole"
          MaxSessionDuration: 43200
          Path: "/"
          AssumeRolePolicyDocument: 
            Version: "2012-10-17"
            Statement: 
            - Effect: "Allow"
              Principal:
                AWS: "[ACCOUNT ARN]"
              Action: "sts:AssumeRole"

      KadaAthenaPolicy: 
        Type: 'AWS::IAM::Policy'
        Properties:
          PolicyName: root
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: 
                  - athena:BatchGetQueryExecution
                  - athena:GetQueryExecution
                  - athena:GetQueryResults
                  - athena:GetQueryResultsStream
                  - athena:ListQueryExecutions
                  - athena:StartQueryExecution
                  - athena:ListWorkGroups
                  - athena:ListDataCatalogs
                  - athena:ListDatabases
                  - athena:ListTableMetadata
                Resource: '*'
              - Effect: Allow
                Action: 
                  - glue:GetDatabase
                  - glue:GetDatabases 
                  - glue:GetTable
                  - glue:GetTables
                  - glue:GetPartition
                  - glue:GetPartitions
                Resource: '*'
              - Effect: Allow
                Action: 
                  - s3:GetBucketLocation
                  - s3:GetObject
                  - s3:ListBucket
                  - s3:ListBucketMultipartUploads
                  - s3:ListMultipartUploadParts
                  - s3:AbortMultipartUpload
                  - s3:PutObject
                  - s3:PutBucketPublicAccessBlock
                  - s3:DeleteObject
                Resource:
                  - arn:aws:s3:::[ATHENA RESULTS BUCKET NAME]
          Roles:
            - !Ref KadaAthenaRole

Alternatively, the following managed policy will also provide the necessary permissions for the collector - <https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonAthenaFullAccess.html>

    aws iam attach-role-policy \
        --role-name YOUR_ROLE_NAME \
        --policy-arn arn:aws:iam::aws:policy/AmazonAthenaFullAccess

After this step you should have the following information

* Athena User

* Role

* Key

* Secret

* Athena S3 bucket location

*** ** * ** ***

## Step 2: Create the Source in K

Create an Athena source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File system" option**

* Give the source a **Name** - e.g. Athena Production

* Add the **Host name** for the Athena Server

* Click **Finish Setup**

*** ** * ** ***

## Step 3: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

1. Go to Platform Settings - Settings. Note down the value of this setting:

   * If using Azure: **storage_azure_storage_account**

   * If using AWS:

     * **storage_root_folder** - the AWS s3 bucket

     * **storage_aws_region** - the region where the AWS s3 bucket is hosted

2. Go to Sources - Edit the Source you have configured. Note down the **landing directory** in the About this Source section.

To connect to the landing directory you will need:

* If using Azure: a **SAS token** to push data to the landing directory. Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

* If using AWS:

  * An **Access key and Secret** . Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

  * OR provide your IAM role to KADA Support to provision access.

*** ** * ** ***

## Step 4: Install the Collector

It is recommended to use a python environment such as **pyenv** or **pipenv** if you are not intending to install this package at the system level.

Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.

You can download the Latest Core Library and Athena whl via **Platform Settings → Sources** → **Download Collectors**

Run the following command to install the collector

    pip install kada_collectors_extractors_<version>-none-any.whl

You will also need to install the common library kada_collectors_lib for this collector to function properly.

    pip install kada_collectors_lib-<version>-none-any.whl

Under the covers this uses boto3 and may have OS dependencies see <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html>

*** ** * ** ***

## Step 5: Configure the Collector

The collector requires a set of parameters to connect to and extract metadata from Athena  

|  **FIELD**  | **FIELD TYPE** |                                         **DESCRIPTION**                                          |           **EXAMPLE**           |
|-------------|----------------|--------------------------------------------------------------------------------------------------|---------------------------------|
| key         | string         | Key for the AWS user                                                                             | "xcvsdsdfsdf"                   |
| secret      | string         | Secret for the AWS user                                                                          | "sgsdfdsfg"                     |
| server      | string         | This is the host that was onboarded in K for Athena                                              | "athena.cloud"                  |
| bucket      | string         | Bucket location to temporarily store Athena query results                                        | "s3://mybucket/myathenaresults" |
| catalogs    | list\<string\> | List of catalogs to extract from Athena                                                          | \["AwsDataCatalog"\]            |
| region      | string         | Set the region for AWS for where Athena exists                                                   | ap-southeast-2                  |
| role        | string         | If your access requires role assumption, place the full arn value here, otherwise leave it blank | ""                              |
| output_path | string         | Absolute path to the output location where files are to be written                               | "/tmp/output"                   |
| mask        | boolean        | To enable masking or not                                                                         | true                            |
| compress    | boolean        | To gzip the output or not                                                                        | true                            |

**kada_athena_extractor_config.json**

    {
        "key": "",
        "secret": "",
        "server": "athena",
        "bucket": "s3://examplebucket/examplefolder",
        "catalogs": ["AwsDataCatalog"],
        "region": "ap-southeast-2",
        "role": "",
        "output_path": "/tmp/output",
        "mask": true,
        "compress": true
    }

*** ** * ** ***

## Step 6: Run the Collector

This is the wrapper script: **kada_athena_extractor.py**

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.athena import Extractor

    get_generic_logger('root')

    _type = 'athena'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA Athena Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename)
    parser.add_argument('--name', '-n', dest='name', default=_type)
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()
    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

*** ** * ** ***

## Step 7: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

**High Water Mark File**

A high water mark file is created called **athena_hwm.txt**.

Refer to [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md) for more information

*** ** * ** ***

## Step 8: Push the Extracts to K

Once the files have been validated, you can push the files to the K landing directory.

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

The following example is how you can orchestrate the Tableau collector using Airflow and push the files to K hosted on Azure. The code is not expected to be used as-is but as a template for your own DAG.
Python

    # built-in
    import os

    # Installed
    from airflow.operators.python_operator import PythonOperator
    from airflow.models.dag import DAG
    from airflow.operators.dummy import DummyOperator
    from airflow.utils.dates import days_ago
    from airflow.utils.task_group import TaskGroup

    from plugins.utils.azure_blob_storage import AzureBlobStorage

    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.tableau import Extractor

    # To be configured by the customer.
    # Note variables may change if using a different object store.
    KADA_SAS_TOKEN = os.getenv("KADA_SAS_TOKEN")
    KADA_CONTAINER = ""
    KADA_STORAGE_ACCOUNT = ""
    KADA_LANDING_PATH = "lz/tableau/landing"
    KADA_EXTRACTOR_CONFIG = {
        "server_address": "http://tabserver",
        "username": "user",
        "password": "password",
        "sites": [],
        "db_host": "tabserver",
        "db_username": "repo_user",
        "db_password": "repo_password",
        "db_port": 8060,
        "db_name": "workgroup",
        "meta_only": False,
        "retries": 5,
        "dry_run": False,
        "output_path": "/set/to/output/path",
        "mask": True,
        "mapping": {}
    }

    # To be implemented by the customer.
    # Upload to your landing zone storage.
    # Change '.csv' to '.csv.gz' if you set compress = true in the config
    def upload():
      output = KADA_EXTRACTOR_CONFIG['output_path']
      for filename in os.listdir(output):
          if filename.endswith('.csv'):
            file_to_upload_path = os.path.join(output, filename)

            AzureBlobStorage.upload_file_sas_token(
                client=KADA_SAS_TOKEN,
                storage_account=KADA_STORAGE_ACCOUNT,
                container=KADA_CONTAINER,
                blob=f'{KADA_LANDING_PATH}/{filename}',
                local_path=file_to_upload_path
            )

    with DAG(dag_id="taskgroup_example", start_date=days_ago(1)) as dag:

        # To be implemented by the customer.
        # Retrieve the timestamp from the prior run
        start_hwm = 'YYYY-MM-DD HH:mm:SS'
        end_hwm = 'YYYY-MM-DD HH:mm:SS' # timestamp now

        ext = Extractor(**KADA_EXTRACTOR_CONFIG)

        start = DummyOperator(task_id="start")

        with TaskGroup("taskgroup_1", tooltip="extract tableau and upload") as extract_upload:
            task_1 = PythonOperator(
                task_id="extract_tableau",
                python_callable=ext.run,
                op_kwargs={"start_hwm": start_hwm, "end_hwm": end_hwm},
                provide_context=True,
            )

            task_2 = PythonOperator(
                task_id="upload_extracts",
                python_callable=upload,
                op_kwargs={},
                provide_context=True,
            )

            # To be implemented by the customer.
            # Timestamp needs to be saved for next run
            task_3 = DummyOperator(task_id='save_hwm')

        end = DummyOperator(task_id='end')

        start >> extract_upload >> end

Last updated: March 21, 2026

---
language: "en"
---
# Athena (via Collector method) - v3.1.0

## About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

### Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it to pull data for security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pulls the latest metadata

3. Storing and pushing the extracts to your K instance

*** ** * ** ***

## Pre-Requisites

**Collector Server Minimum Requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**Athena Requirements**

* Access to**Athena**

*** ** * ** ***

## Step 1: Establish Athena Access

It is advised you create a new Role and a separate s3 bucket for the service user provided to KADA and have a policy that allows the below, see [Identity and access management in Athena - Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/security-iam-athena.html)

The service user/account/role will require permissions to the following

1. Execute queries against Athena with access to the INFORMATION_SCHEMA in particular the following tables:

   1. information_schema.views

   2. information_schema.tables

   3. information_schema.columns

2. Executing queries in Athena requires an s3 bucket to temporarily store results. We will also require the policy to allow Read Write Listing access to objects within that bucket, conversely, the bucket must also have policy to allow to do the same.

3. Call the following Athena APIs (Note that access to Athena metadata through the below APIs will also require access to the Glue catalog).

   1. [BatchGetQueryExecutions](https://docs.aws.amazon.com/athena/latest/APIReference/API_BatchGetQueryExecution.html)

   2. [GetQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html)

   3. [GetQueryResults](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html)

   4. [ListQueryExecutions](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListQueryExecutions.html)

   5. [StartQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_StartQueryExecution.html)

   6. [ListWorkGroups](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListWorkGroups.html)

   7. [ListDataCatalogs](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListDataCatalogs.html)

   8. [ListDatabases](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListDatabases.html)

   9. [ListTableMetadata](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListTableMetadata.html)

4. The service user/account/role will need permissions to access all workgroups to be able to extract all data, if you omit workgroups, that information will not be extracted and you may not see the complete picture in K.

5. See [IAM policies for accessing workgroups - Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/workgroups-iam-policy.html) on how to add policy entries to have fine grain control at the workgroup level. Note that the extractor runs queries on Athena, If you do choose to restrict workgroup access, ensure that Query based actions (e.g. StartQueryExecution) are allowed for the workgroup the service user/account/role is associated to.

Note that user usage will be associated to the workgroup level rather than individual users, these workgroups are published as users in K in the form "athena_workgroup_\<name\>"

Example Role Policy to allow Athena Access with least privileges for actions, this example allows the **ACCOUNT ARN** to assume the role. Note the variables **ATHENA RESULTS BUCKET NAME.**You may also choose to just assign the policy directly to a new user and use that user without assuming roles. In the scenario you do wish to assume a role, please note down the role ARN to be used when onboarding/extracting.

    AWSTemplateFormatVersion: "2010-09-09"
    Description: 'AWS IAM Role - Athena Access to KADA'
    Resources: 
      KadaAthenaRole: 
        Type: "AWS::IAM::Role"
        Properties: 
          RoleName: "KadaAthenaRole"
          MaxSessionDuration: 43200
          Path: "/"
          AssumeRolePolicyDocument: 
            Version: "2012-10-17"
            Statement: 
            - Effect: "Allow"
              Principal:
                AWS: "[ACCOUNT ARN]"
              Action: "sts:AssumeRole"

      KadaAthenaPolicy: 
        Type: 'AWS::IAM::Policy'
        Properties:
          PolicyName: root
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: 
                  - athena:BatchGetQueryExecution
                  - athena:GetQueryExecution
                  - athena:GetQueryResults
                  - athena:GetQueryResultsStream
                  - athena:ListQueryExecutions
                  - athena:StartQueryExecution
                  - athena:ListWorkGroups
                  - athena:ListDataCatalogs
                  - athena:ListDatabases
                  - athena:ListTableMetadata
                Resource: '*'
              - Effect: Allow
                Action: 
                  - glue:GetDatabase
                  - glue:GetDatabases 
                  - glue:GetTable
                  - glue:GetTables
                  - glue:GetPartition
                  - glue:GetPartitions
                Resource: '*'
              - Effect: Allow
                Action: 
                  - s3:GetBucketLocation
                  - s3:GetObject
                  - s3:ListBucket
                  - s3:ListBucketMultipartUploads
                  - s3:ListMultipartUploadParts
                  - s3:AbortMultipartUpload
                  - s3:PutObject
                  - s3:PutBucketPublicAccessBlock
                  - s3:DeleteObject
                Resource:
                  - arn:aws:s3:::[ATHENA RESULTS BUCKET NAME]
          Roles:
            - !Ref KadaAthenaRole

Alternatively, the following managed policy will also provide the necessary permissions for the collector - <https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonAthenaFullAccess.html>

    aws iam attach-role-policy \
        --role-name YOUR_ROLE_NAME \
        --policy-arn arn:aws:iam::aws:policy/AmazonAthenaFullAccess

After this step you should have the following information

* Athena User

* Role

* Key

* Secret

* Athena S3 bucket location

*** ** * ** ***

## Step 2: Create the Source in K

Create an Athena source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File system" option**

  ![image-20221106-104455.png](https://docs.kada.ai/__attachments/a_5f30893a847a292d25981c97d6f3a5cd8ecb621ca41dd612c0d3c1dbf99b8476/image-20221106-104455.png?cb=c68c7d4d0eb7814d55fa833955d85bdc)

* Give the source a **Name** - e.g. Athena Production

* Add the **Host name** for the Athena Server

* Click **Finish Setup**

*** ** * ** ***

## Step 3: Getting Access to the Source Landing Directory

*** ** * ** ***

## Step 4: Install the Collector

It is recommended to use a python environment such as **pyenv** or **pipenv** if you are not intending to install this package at the system level.

Some python packages also have dependencies on the OS level packages, so you may be required to install additional OS packages if the below fails to install.

You can download the Latest Core Library and Athena whl via **Platform Settings → Sources** → **Download Collectors**  
![image-20260715-133348.png](https://docs.kada.ai/__attachments/a_48b018752fd22f188e6cd01f8571e276088cde3e24145650c11df42ccd56c596/image-20260715-133348.png?cb=91b43aca3defeef5504bf531e5277a3e)

Run the following command to install the collector

    pip install kada_collectors_extractors_<version>-none-any.whl

You will also need to install the common library kada_collectors_lib for this collector to function properly.

    pip install kada_collectors_lib-<version>-none-any.whl

Under the covers this uses boto3 and may have OS dependencies see <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html>

*** ** * ** ***

## Step 5: Configure the Collector

The collector requires a set of parameters to connect to and extract metadata from Athena  

|    **FIELD**    | **FIELD TYPE** |                                                                                             **DESCRIPTION**                                                                                             |           **EXAMPLE**           |
|-----------------|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------|
| key             | string         | Key for the AWS user                                                                                                                                                                                    | "xcvsdsdfsdf"                   |
| secret          | string         | Secret for the AWS user                                                                                                                                                                                 | "sgsdfdsfg"                     |
| server          | string         | This is the host that was onboarded in K for Athena                                                                                                                                                     | "athena.cloud"                  |
| bucket          | string         | Bucket location to temporarily store Athena query results, the extractor will use the user to execute queries and store results in this bucket location, it should be the full path starting with s3:// | "s3://mybucket/myathenaresults" |
| catalogs        | list\<string\> | List of catalogs to extract from Athena, most cases this is only AwsDataCatalog unless you have self managed catalogs.                                                                                  | \["AwsDataCatalog"\]            |
| region          | string         | Set the region for AWS for where Athena exists                                                                                                                                                          | ap-southeast-2                  |
| role            | string         | If your access requires role assumption, place the full arn value here, otherwise leave it blank                                                                                                        | ""                              |
| output_path     | string         | Absolute path to the output location where files are to be written                                                                                                                                      | "/tmp/output"                   |
| mask            | boolean        | To enable masking or not                                                                                                                                                                                | true                            |
| masking_workers | integer        | Number of parallel workers to use for masking                                                                                                                                                           | 2                               |
| compress        | boolean        | To gzip the output or not                                                                                                                                                                               | true                            |
| meta_only       | boolean        | To extract metadata only                                                                                                                                                                                | true                            |
| events_only     | boolean        | To extract events only                                                                                                                                                                                  | true                            |
| chunk_hours     | integer        | Number of hours to incrementally chunk the extract of events out in                                                                                                                                     | 24                              |

These parameters can be added directly into the run or you can use pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.

**kada_athena_extractor_config.json**

    {
        "key": "",
        "secret": "",
        "server": "athena",
        "bucket": "s3://examplebucket/examplefolder",
        "catalogs": ["AwsDataCatalog"],
        "region": "ap-southeast-2",
        "role": "",
        "output_path": "/tmp/output",
        "mask": true,
        "masking_workers": 1
        "compress": true,
        "meta_only": true,
        "events_only": true
    }

*** ** * ** ***

## Step 6: Run the Collector

The following code is an example of how to run the extractor. You may need to uplift this code to meet any code standards at your organisation.

This can be executed in any python environment where the whl has been installed. It will produce and read a high water mark file from the same directory as the execution called **athena_hwm.txt** and produce files according to the configuration JSON.

This is the wrapper script: **kada_athena_extractor.py**

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.athena import Extractor

    get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

    _type = 'athena'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA Athena Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
    parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()
    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

**Advance options:**

If you wish to maintain your own high water mark files elsewhere you can use the above section's script as a guide on how to call the extractor. The configuration file is simply the keyword arguments in JSON format.

If you are handling external arguments of the runner yourself, you'll need to consider additional items for the run method.

Refer to this document for more information [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md)

    from kada_collectors.extractors.snowflake import Extractor

    kwargs = {my args} # However you choose to construct your args
    hwm_kwrgs = {"start_hwm": "end_hwm": } # The hwm values

    ext = Extractor(**kwargs)
    ext.run(**hwm_kwrgs)

*** ** * ** ***

    class Extractor(key: str = None, secret: str = None, server: str = None, \
          bucket: str = None, catalogs: list = ['AwsDataCatalog'], \
          region: str = 'ap-southeast-2', role: str = None, \
          output_path: str = './output', mask: bool = False, compress: bool = False) -> None

key: AWS Access Key.

secret: AWS Secret.

region: Region.

server: Athena host that was onboarded on K.

role: AWS Role ARN if required to assume a role. bucket: s3 bucket used to temporarily store results in the form s3://xxx.

catalogs: list of Catalogs from Athena to extract, by default this is just AwsDataCatalog.

output_path: full or relative path to where the outputs should go

compress: To gzip output files or not

*** ** * ** ***

## Step 7: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated. These files will appear in the output_path directory you set in the configuration details

**High Water Mark File**

A high water mark file is created in the same directory as the execution called **athena_hwm.txt** and produce files according to the configuration JSON. This file is only produced if you call the publish_hwm method.

If you want prefer file managed hwm, you can edit the location of the hwn by following these instructions [Collector Integration General Notes \| Storing High Water Marks (HWM)](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md#Storing-High-Water-Marks-(HWM))

*** ** * ** ***

## Step 8: Push the Extracts to K

Once the files have been validated, you can push the files to the K landing directory.

You can use [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/) if you want to initially do this manually. You can push the files using python as well (see Airflow example below)

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

Last updated: July 15, 2026

---
language: "en"
---
# Athena (via Direct Connect method)

This page will walkthrough the setup of Athena in K using the direct connect method.

## Integration details

|       **Scope**        | **Included** | **Comments** |
|------------------------|--------------|--------------|
| Metadata               | YES          |              |
| Lineage                | YES          |              |
| Usage                  | YES          |              |
| Sensitive Data Scanner | N/A          |              |

*** ** * ** ***

## Step 1: Establish Athena Access

It is advised you create a new Role and a separate s3 bucket for the service user provided to KADA and have a policy that allows the below. See [Identity and access management in Athena - Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/security-iam-athena.html).

The service user/account/role will require permissions to the following:

1. Execute queries against Athena with access to the INFORMATION_SCHEMA, in particular the following tables:

   * information_schema.views

   * information_schema.tables

   * information_schema.columns

2. Executing queries in Athena requires an s3 bucket to temporarily store results. We will also require the policy to allow Read Write Listing access to objects within that bucket; conversely, the bucket must also have policy to allow to do the same.

3. Call the following Athena APIs (Note that access to Athena metadata through the below APIs will also require access to the Glue catalog):

   * [BatchGetQueryExecutions](https://docs.aws.amazon.com/athena/latest/APIReference/API_BatchGetQueryExecution.html)

   * [GetQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryExecution.html)

   * [GetQueryResults](https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html)

   * [ListQueryExecutions](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListQueryExecutions.html)

   * [StartQueryExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_StartQueryExecution.html)

   * [ListWorkGroups](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListWorkGroups.html)

   * [ListDataCatalogs](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListDataCatalogs.html)

   * [ListDatabases](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListDatabases.html)

   * [ListTableMetadata](https://docs.aws.amazon.com/athena/latest/APIReference/API_ListTableMetadata.html)

4. The service user/account/role will need permissions to access all workgroups to be able to extract all data. If you omit workgroups, that information will not be extracted and you may not see the complete picture in K.

5. See [IAM policies for accessing workgroups - Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/workgroups-iam-policy.html) on how to add policy entries to have fine grain control at the workgroup level. Note that the extractor runs queries on Athena. If you do choose to restrict workgroup access, ensure that Query based actions (e.g. StartQueryExecution) are allowed for the workgroup the service user/account/role is associated to.

Note that user usage will be associated to the workgroup level rather than individual users. These workgroups are published as users in K in the form `athena_workgroup_<name>`.

**Example Role Policy (AWS CloudFormation template)** to allow Athena Access with least privileges. This example allows the **ACCOUNT ARN** to assume the role. Note the variable **ATHENA RESULTS BUCKET NAME**. You may also choose to just assign the policy directly to a new user and use that user without assuming roles. If you do wish to assume a role, please note down the role ARN to be used when onboarding/extracting.
YAML

    AWSTemplateFormatVersion: "2010-09-09"
    Description: 'AWS IAM Role - Athena Access to KADA'
    Resources: 
      KadaAthenaRole: 
        Type: "AWS::IAM::Role"
        Properties: 
          RoleName: "KadaAthenaRole"
          MaxSessionDuration: 43200
          Path: "/"
          AssumeRolePolicyDocument: 
            Version: "2012-10-17"
            Statement: 
            - Effect: "Allow"
              Principal:
                AWS: "[ACCOUNT ARN]"
              Action: "sts:AssumeRole"

      KadaAthenaPolicy: 
        Type: 'AWS::IAM::Policy'
        Properties:
          PolicyName: root
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: 
                  - athena:BatchGetQueryExecution
                  - athena:GetQueryExecution
                  - athena:GetQueryResults
                  - athena:GetQueryResultsStream
                  - athena:ListQueryExecutions
                  - athena:StartQueryExecution
                  - athena:ListWorkGroups
                  - athena:ListDataCatalogs
                  - athena:ListDatabases
                  - athena:ListTableMetadata
                Resource: '*'
              - Effect: Allow
                Action: 
                  - glue:GetDatabase
                  - glue:GetDatabases 
                  - glue:GetTable
                  - glue:GetTables
                  - glue:GetPartition
                  - glue:GetPartitions
                Resource: '*'
              - Effect: Allow
                Action: 
                  - s3:GetBucketLocation
                  - s3:GetObject
                  - s3:ListBucket
                  - s3:ListBucketMultipartUploads
                  - s3:ListMultipartUploadParts
                  - s3:AbortMultipartUpload
                  - s3:PutObject
                  - s3:PutBucketPublicAccessBlock
                  - s3:DeleteObject
                Resource:
                  - arn:aws:s3:::[ATHENA RESULTS BUCKET NAME]
          Roles:
            - !Ref KadaAthenaRole

Alternatively, the following managed policy will also provide the necessary permissions for the collector:
Bash

    aws iam attach-role-policy \
        --role-name YOUR_ROLE_NAME \
        --policy-arn arn:aws:iam::aws:policy/AmazonAthenaFullAccess

See also: [AmazonAthenaFullAccess managed policy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonAthenaFullAccess.html)  
After this step you should have the following information:

* Athena User

* Role

* Key

* Secret

* Athena S3 bucket location

*** ** * ** ***

## Step 2: Create the Source in K

Create an Athena source in K.

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Click **Add Source** and select **Athena**

  ![image-20230530-111137.png](https://docs.kada.ai/__attachments/a_02a6a202317c058c06a1302230279eba431f09709bb9020474dbf1f349830026/image-20230530-111137.png?cb=4d1a12358023502e51b6c541db8cdc96)
* Select **Direct Connect** and add your Athena details

  * **Name:** Give the Athena source a name in K.

  * **Host:** Enter a hostname for your Athena instance

  * **Region:** Set the region for AWS for where Athena exists e.g. ap-southeast-2

  * **Athena Results bucket:** Bucket location used to temporarily store Athena query results. Use the full path starting with s3://

* Add **Connection Details** and click **Save \& Next**

  * Assume Role: Add the **Role** from Step 1

  * Key: Add the **Key** from Step 1

  * Secret: Add the **Secret** from Step 1

* **Test your connection** and click **Next**

* Click **Finish Setup**

*** ** * ** ***

## Step 3: Schedule Athena source load

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Locate your new Athena Source and click on the **Schedule Settings** (clock) icon to set the schedule

*** ** * ** ***

## Step 4: Manually run an ad hoc load to test Athena

Last updated: July 17, 2026

---
language: "en"
---
# Automate Data Tagging

Updated in Version 6.0

K removes the manual nature of tagging data by automating the process through Linking Rules. This minimises data governance gaps and ensures consistent, accurate classification of data assets at scale.  
![image-20260726-121431.png](https://docs.kada.ai/__attachments/a_c0c4406eb580c3ba2880fb0117ea4cf6de7105199ef93cb205a94633c9ae3ca6/image-20260726-121431.png?cb=840cbb2ca531888645114df2bafdfbb4)

*** ** * ** ***

## How It Works

Administrators configure tagging rules based on metadata attributes such as column names, data types, source systems, or PII detection results. When K processes an asset that matches a rule, the corresponding tag is automatically applied --- no manual intervention required.

Examples of automated tagging rules:

* Any column named `email` → apply tag: `PII: Email`

* Any table in schema `finance.*` → apply tag: `Domain: Finance`

* Any asset flagged by PII scanner → apply tag: `Sensitive`

*** ** * ** ***

## Benefits

* **Consistency** --- rules ensure the same tags are applied uniformly across assets, regardless of who created them

* **Scale** --- thousands of assets can be tagged in a single run, far faster than manual effort

* **Reduced governance gaps** --- new assets are automatically classified as soon as they are ingested into K

* **Auditability** --- automated tags are traceable back to the rules that created them

*** ** * ** ***

## Related

* [Detect PII](https://docs.kada.ai/k-knowledge-base/detect-pii.md)

* [Governance Dashboards](https://docs.kada.ai/k-knowledge-base/governance-dashboards.md)

Last updated: August 15, 2026

---
language: "en"
---
# AWS RDS Postgres (via Collector method)

This page outlines the AWS RDS Postgres Collector versions that are available.

We always recommend using the latest Collector Method to ensure that you can access the latest features.

*** ** * ** ***

## Integration details

|       **Scope**        | **Included** | **Comments** |
|------------------------|--------------|--------------|
| Metadata               | YES          |              |
| Lineage                | YES          |              |
| Usage                  | No           |              |
| Sensitive Data Scanner | No           |              |

*** ** * ** ***

## AWS RDS Postgres Version History

|                             **Version Number**                              | **Release Month** | **K Platform Compatibility** | **Collector Core Library Compatibility** |                            **Release changes**                             |
|-----------------------------------------------------------------------------|-------------------|------------------------------|------------------------------------------|----------------------------------------------------------------------------|
| [V3.0.0](https://docs.kada.ai/k-knowledge-base/aws-rds-postgres-via-collector-method-v3-0-0.md) | April 2024        | 5.33                         | 1.1 - 1.3                                | Added override to remove Auth_id table that does not exist in RDS Postgres |

Last updated: March 15, 2026

---
language: "en"
---
# AWS RDS Postgres (via Collector method) - v3.0.0

## About Collectors

Collectors are extractors that are developed and managed by you (A customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

**Why you should use a Collector**

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it pull data for Security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pull the latest metadata

3. Storing and pushing the extracts to your K instance.

*** ** * ** ***

## Pre-requisites

**Collector Server Minimum Requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**AWS RDS Postgres Requirements**

The user used for the extractor will need access to a number of pg_catalog tables outlined below

**PG Catalog**

Generally all users should have access to the pg_catalog tables on DB creation. In the event the user doesn't have access, explicit grants will need to be done per new DB in Postgres.
SQL

    GRANT USAGE ON SCHEMA pg_catalog TO <kada user>;
    GRANT SELECT ON ALL TABLES IN SCHEMA pg_catalog TO <kada user>;

The user used for the extraction must also be able to connect to the the databases needed for extraction.

**PG Tables**

These tables are per database in Postgres

* pg_class

* pg_namespace

* pg_proc

* pg_database

* pg_language

* pg_type

* pg_collation

* pg_depend

* pg_sequence

* pg_constraint

* pg_auth_members

**Databases**

* All other databases that you want onboarded

Note that visibility of entries in these tables will depend on if the user has SELECT access to the table, so make sure SELECT is granted to the \<kada user\> for all tables within the database.

1. SQL

   ```

   ```

GRANT SELECT ON ALL TABLES IN SCHEMA \<schema\> TO \<kada user\>

    2. ```sql
    ALTER DEFAULT PRIVILEGES IN SCHEMA <schema> public GRANT SELECT ON TABLES TO <kada user>

*** ** * ** ***

## Step 1: Create the Source in K

Create a Postgres source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File" option**

* Give the source a **Name** - e.g. Postgres Production

* Add the **Host name** for the Postgres Server

* Click **Finish Setup**

*** ** * ** ***

## Step 2: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

1. Go to Platform Settings - Settings. Note down the value of this setting:

   * If using Azure: **storage_azure_storage_account**

   * If using AWS:

     * **storage_root_folder** - the AWS s3 bucket

     * **storage_aws_region** - the region where the AWS s3 bucket is hosted

2. Go to Sources - Edit the Source you have configured. Note down the **landing directory** in the About this Source section.

To connect to the landing directory you will need:

* If using Azure: a **SAS token** to push data to the landing directory. Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

* If using AWS:

  * An **Access key and Secret** . Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

  * OR provide your IAM role to KADA Support to provision access.

*** ** * ** ***

## Step 3: Install the Collector

You can download the latest Core Library via **Platform Settings → Sources** → **Download Collectors**

Run the following command to install the collector.

    pip install kada_collectors_extractors_<version>-none-any.whl

You will also need to install the common library kada_collectors_lib for this collector to function properly.

    pip install kada_collectors_lib-<version>-none-any.whl

*** ** * ** ***

## Step 4: Configure the Collector

|  **FIELD**  | **FIELD TYPE** |                      **DESCRIPTION**                      |         **EXAMPLE**          |
|-------------|----------------|-----------------------------------------------------------|------------------------------|
| host        | string         | Postgres host as per what was onboarded in the K platform | "example.postgres.localhost" |
| server      | string         | Postgres host to establish a connection                   | "example.postgres.localhost" |
| username    | string         | Username to log into Postgres                             | "postgres_user"              |
| password    | string         | Password to log into the Postgres                         |                              |
| databases   | list\<string\> | A list of databases to extract from Postgres              | \["dwh", "adw"\]             |
| port        | integer        | Postgres port, general default is 5432                    | 5432                         |
| output_path | string         | Absolute path to the output location                      | "/tmp/output"                |
| mask        | boolean        | To enable masking or not                                  | true                         |
| compress    | boolean        | To gzip the output or not                                 | true                         |
| meta_only   | boolean        | To extract metadata only or not                           | true                         |

**kada_postgres_extractor_config.json**
JSON

    {
        "host": "",
        "server": "",
        "username": "",
        "password": "",
        "databases": [],
        "port": 5432,
        "output_path": "/tmp/output",
        "mask": true,
        "compress": true,
        "meta_only": true
    }

*** ** * ** ***

## Step 5: Run the Collector

This is the wrapper script: **kada_postgres_extractor.py**
Python

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.postgres import Extractor

    get_generic_logger('root')

    _type = 'postgres'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA Postgres Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename)
    parser.add_argument('--name', '-n', dest='name', default=_type)
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()

    disable_roles_sql = """SELECT DISTINCT 'USER' AS "OBJECT_TYPE", '' AS "OBJECT_ID", '' AS "USER", '' AS "ROLE" where 1 = 2"""
    ext.overwrite_sql('ROLES_SQL', disable_roles_sql)

    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

*** ** * ** ***

## Step 6: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

**High Water Mark File**

A high water mark file is created called **postgres_hwm.txt**.

Refer to [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md) for more information.

*** ** * ** ***

## Step 7: Push the Extracts to K

Once the files have been validated, you can push the files to the [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md).

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

The following example is how you can orchestrate the Tableau collector using Airflow and push the files to K hosted on Azure. The code is not expected to be used as-is but as a template for your own DAG.
Python

    # built-in
    import os

    # Installed
    from airflow.operators.python_operator import PythonOperator
    from airflow.models.dag import DAG
    from airflow.operators.dummy import DummyOperator
    from airflow.utils.dates import days_ago
    from airflow.utils.task_group import TaskGroup

    from plugins.utils.azure_blob_storage import AzureBlobStorage

    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.tableau import Extractor

    # To be configured by the customer.
    # Note variables may change if using a different object store.
    KADA_SAS_TOKEN = os.getenv("KADA_SAS_TOKEN")
    KADA_CONTAINER = ""
    KADA_STORAGE_ACCOUNT = ""
    KADA_LANDING_PATH = "lz/tableau/landing"
    KADA_EXTRACTOR_CONFIG = {
        "server_address": "http://tabserver",
        "username": "user",
        "password": "password",
        "sites": [],
        "db_host": "tabserver",
        "db_username": "repo_user",
        "db_password": "repo_password",
        "db_port": 8060,
        "db_name": "workgroup",
        "meta_only": False,
        "retries": 5,
        "dry_run": False,
        "output_path": "/set/to/output/path",
        "mask": True,
        "mapping": {}
    }

    # To be implemented by the customer.
    # Upload to your landing zone storage.
    # Change '.csv' to '.csv.gz' if you set compress = true in the config
    def upload():
      output = KADA_EXTRACTOR_CONFIG['output_path']
      for filename in os.listdir(output):
          if filename.endswith('.csv'):
            file_to_upload_path = os.path.join(output, filename)

            AzureBlobStorage.upload_file_sas_token(
                client=KADA_SAS_TOKEN,
                storage_account=KADA_STORAGE_ACCOUNT,
                container=KADA_CONTAINER,
                blob=f'{KADA_LANDING_PATH}/{filename}',
                local_path=file_to_upload_path
            )

    with DAG(dag_id="taskgroup_example", start_date=days_ago(1)) as dag:

        # To be implemented by the customer.
        # Retrieve the timestamp from the prior run
        start_hwm = 'YYYY-MM-DD HH:mm:SS'
        end_hwm = 'YYYY-MM-DD HH:mm:SS' # timestamp now

        ext = Extractor(**KADA_EXTRACTOR_CONFIG)

        start = DummyOperator(task_id="start")

        with TaskGroup("taskgroup_1", tooltip="extract tableau and upload") as extract_upload:
            task_1 = PythonOperator(
                task_id="extract_tableau",
                python_callable=ext.run,
                op_kwargs={"start_hwm": start_hwm, "end_hwm": end_hwm},
                provide_context=True,
            )

            task_2 = PythonOperator(
                task_id="upload_extracts",
                python_callable=upload,
                op_kwargs={},
                provide_context=True,
            )

            # To be implemented by the customer.
            # Timestamp needs to be saved for next run
            task_3 = DummyOperator(task_id='save_hwm')

        end = DummyOperator(task_id='end')

        start >> extract_upload >> end

Last updated: March 21, 2026

---
language: "en"
---
# Azure Data Factory (via Collector method)

This page outlines the Azure Data Factory Collector versions that are available.

We always recommend using the latest Collector Method to ensure that you can access the latest features.

*** ** * ** ***

## Integration details

|       **Scope**        | **Included** | **Comments** |
|------------------------|--------------|--------------|
| Metadata               | YES          | See below    |
| Lineage                | YES          |              |
| Usage                  | YES          |              |
| Sensitive Data Scanner | N/A          |              |

**Known Azure Data Factory Collector limitations**

* Not all sources and destinations are included in the metadata extraction. Improvements are planned to provide wider coverage

* Sources currently implemented: Snowflake

*** ** * ** ***

## Azure Data Factory Version History

|                            **Version Number**                             | **Release Month** | **K Platform Compatibility** | **Collector Core Library Compatibility** |        **Release changes**        |
|---------------------------------------------------------------------------|-------------------|------------------------------|------------------------------------------|-----------------------------------|
| [V3.1](https://docs.kada.ai/k-knowledge-base/azure-data-factory-via-collector-method-v3-1.md) | September 2023    | 5.23+                        | 1.1+                                     | Updated additional metadata feeds |

Last updated: March 15, 2026

---
language: "en"
---
# Azure Data Factory (via Collector method) - v3.1

## About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

### Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it to pull data for security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pulls the latest metadata

3. Storing and pushing the extracts to your K instance

*** ** * ** ***

## Pre-requisites

**Collector Server Minimum Requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**Azure Data Factory Requirements**

* Access to Azure Data Factory

*** ** * ** ***

## **Step 1: Create the Source in K**

Create a Azure Data Factory source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File" option**

* Give the source a **Name** - e.g. Azure Data Factory Production

* Add the **Host name** for the Azure Data Factory Server

* Click **Finish Setup**

*** ** * ** ***

## Step 2: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

1. Go to Platform Settings - Settings. Note down the value of this setting:

   * If using Azure: **storage_azure_storage_account**

   * If using AWS:

     * **storage_root_folder** - the AWS s3 bucket

     * **storage_aws_region** - the region where the AWS s3 bucket is hosted

2. Go to Sources - Edit the Source you have configured. Note down the **landing directory** in the About this Source section.

To connect to the landing directory you will need:

* If using Azure: a **SAS token** to push data to the landing directory. Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

* If using AWS:

  * An **Access key and Secret** . Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

  * OR provide your IAM role to KADA Support to provision access.

*** ** * ** ***

## Step 3: Install the Collector

You can download the latest Core Library and whl via **Platform Settings → Sources** → **Download Collectors**

Run the following command to install the collector.

    pip install kada_collectors_extractors_<version>-none-any.whl

You will also need to install the common library kada_collectors_lib for this collector to function properly.

    pip install kada_collectors_lib-<version>-none-any.whl

*** ** * ** ***

## Step 4: Configure the Collector

|      **FIELD**      | **FIELD TYPE** |                                    **DESCRIPTION**                                    |                       **EXAMPLE**                       |
|---------------------|----------------|---------------------------------------------------------------------------------------|---------------------------------------------------------|
| client              | string         | Onboarded client in Azure to access ADF                                               |                                                         |
| secret              | string         | Onboarded client secret in Azure to access ADF                                        |                                                         |
| tenant              | string         | Tenant ID of where ADF exists                                                         |                                                         |
| subscription_id     | string         | Subscription in Azure which the ADF is associated to                                  |                                                         |
| resource_group_name | string         | Resource group in Azure which the ADF is associated to                                |                                                         |
| factory_name        | string         | The name of the ADF factory                                                           |                                                         |
| output_path         | string         | Absolute path to the output location                                                  | "/tmp/output"                                           |
| mask                | boolean        | To enable masking or not                                                              | true                                                    |
| timeout             | integer        | Timeout in seconds allowed against the ADF APIs                                       | 20                                                      |
| mapping             | json           | Mapping file of data source names against the onboarded host and database name in K   | {"myDSN": {"host": "myhost", "database": "mydatabase"}} |
| compress            | boolean        | To compress the output                                                                | true                                                    |
| active_days         | integer        | The pipeline must have been run within active days from today to be considered active | 60                                                      |

**kada_adf_extractor_config.json**
JSON

    {
        "client": "",
        "secret": "",
        "tenant": "",
        "subscription_id": "",
        "resource_group_name": "",
        "factory_name": "",
        "output_path": "/tmp/output",
        "mask": true,
        "timeout": 20,
        "mapping": {
            "myDSN": {
                "host": "myhost",
                "database": "mydatabase"
            }
        },
        "compress": true,
        "active_days": 60
    }

*** ** * ** ***

## Step 5: Run the Collector

This is the wrapper script: **kada_adf_extractor.py**
Python

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.adf import Extractor

    get_generic_logger('root')

    _type = 'adf'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA ADF Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename)
    parser.add_argument('--name', '-n', dest='name', default=_type)
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()
    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

*** ** * ** ***

## Step 6: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

**High Water Mark File**

A high water mark file is created called **adf_hwm.txt**.

Refer to [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md) for more information.

*** ** * ** ***

## Step 7: Push the Extracts to K

Once the files have been validated, you can push the files to the [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md).

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

The following example is how you can orchestrate the Tableau collector using Airflow and push the files to K hosted on Azure. The code is not expected to be used as-is but as a template for your own DAG.
Python

    # built-in
    import os

    # Installed
    from airflow.operators.python_operator import PythonOperator
    from airflow.models.dag import DAG
    from airflow.operators.dummy import DummyOperator
    from airflow.utils.dates import days_ago
    from airflow.utils.task_group import TaskGroup

    from plugins.utils.azure_blob_storage import AzureBlobStorage

    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.tableau import Extractor

    # To be configured by the customer.
    # Note variables may change if using a different object store.
    KADA_SAS_TOKEN = os.getenv("KADA_SAS_TOKEN")
    KADA_CONTAINER = ""
    KADA_STORAGE_ACCOUNT = ""
    KADA_LANDING_PATH = "lz/tableau/landing"
    KADA_EXTRACTOR_CONFIG = {
        "server_address": "http://tabserver",
        "username": "user",
        "password": "password",
        "sites": [],
        "db_host": "tabserver",
        "db_username": "repo_user",
        "db_password": "repo_password",
        "db_port": 8060,
        "db_name": "workgroup",
        "meta_only": False,
        "retries": 5,
        "dry_run": False,
        "output_path": "/set/to/output/path",
        "mask": True,
        "mapping": {}
    }

    # To be implemented by the customer.
    # Upload to your landing zone storage.
    # Change '.csv' to '.csv.gz' if you set compress = true in the config
    def upload():
      output = KADA_EXTRACTOR_CONFIG['output_path']
      for filename in os.listdir(output):
          if filename.endswith('.csv'):
            file_to_upload_path = os.path.join(output, filename)

            AzureBlobStorage.upload_file_sas_token(
                client=KADA_SAS_TOKEN,
                storage_account=KADA_STORAGE_ACCOUNT,
                container=KADA_CONTAINER,
                blob=f'{KADA_LANDING_PATH}/{filename}',
                local_path=file_to_upload_path
            )

    with DAG(dag_id="taskgroup_example", start_date=days_ago(1)) as dag:

        # To be implemented by the customer.
        # Retrieve the timestamp from the prior run
        start_hwm = 'YYYY-MM-DD HH:mm:SS'
        end_hwm = 'YYYY-MM-DD HH:mm:SS' # timestamp now

        ext = Extractor(**KADA_EXTRACTOR_CONFIG)

        start = DummyOperator(task_id="start")

        with TaskGroup("taskgroup_1", tooltip="extract tableau and upload") as extract_upload:
            task_1 = PythonOperator(
                task_id="extract_tableau",
                python_callable=ext.run,
                op_kwargs={"start_hwm": start_hwm, "end_hwm": end_hwm},
                provide_context=True,
            )

            task_2 = PythonOperator(
                task_id="upload_extracts",
                python_callable=upload,
                op_kwargs={},
                provide_context=True,
            )

            # To be implemented by the customer.
            # Timestamp needs to be saved for next run
            task_3 = DummyOperator(task_id='save_hwm')

        end = DummyOperator(task_id='end')

        start >> extract_upload >> end

Last updated: March 21, 2026

---
language: "en"
---
# Azure Data Factory (via Direct Connect method)

This page will walkthrough the setup of Azure Data Factory in K using the direct connect method.

## Integration details

|       **Scope**        | **Included** | **Comments** |
|------------------------|--------------|--------------|
| Metadata               | YES          | See below    |
| Lineage                | YES          |              |
| Usage                  | YES          |              |
| Sensitive Data Scanner | N/A          |              |

**Known limitations**

* Not all sources and destinations are included in the metadata extraction. Improvements are planned to provide wider coverage.

* Sources Implemented:

  * SNOWFLAKE

*** ** * ** ***

## Step 1) Enabling Azure Data Factory Admin APIs to be accessible to an AD Group

This step is performed by the Azure Data Factory Admin.

* Under **Azure services** click on **Data factories**

  ![image-20221008-121630.png](https://docs.kada.ai/__attachments/a_1d6a9ad1d1871d56649141074fa3f23131b38bed47930f2162a32406b3ff537c/image-20221008-121630.png?cb=8618735aedc6b58a79f7c4c76e71273e)
* Locate the Data Factory that you would like to connect to K

* Click on **Overview** to copy the below details for a later step:

  * **Factory name**

  * **Resource group name**

  * **Subscription ID**

    ![image-20230426-115506.png](/__attachments/a_cd9d461288b39a52baed1bd7e53fcc8b22a01d03203fb42790979eef9f29fa50/image-20230426-115506.png?cb=0cd1b8bb70b9e368f425fd862088dc90)

*** ** * ** ***

## Step 2) Registering Azure Data Factory App in Azure AD

This step is performed by the Azure AD Admin.

* Log in to your company's **Azure Portal** and open the **Azure Active Directory** page

* Select **App Registration** in the side panel and click **New registration**

* Complete the registration form

  * Name: Enter a name for the integration e.g. **KADA Azure Data Factory API Integration**

  * Supported account types: Select **Accounts in this organisation directory only**

  * Redirect URL: Add Web / [https://www.kada.ai](https://www.kada.ai/)

* Click **Register** to complete the registration

* Click on the newly created **KADA Azure Data Factory API Integration** App

* Save the **Application (client) ID** and **Directory (tenant) ID** for use in a later step

* Click on **Endpoints** and save the URL for **OpenID Connect metadata document** for use in a later step

* Select **Certificates \& secrets** in the side panel and click **New client secret**

* Complete the new secret form and save the **Secret Value** for use in a later step

Make sure you send all of the information from Step 1 and Step 2 to the K Admin so that they can complete step 4.

* Factory name

* Resource group name

* Subscription ID

* Application (client) ID

* Directory (tenant) ID

* Secret Value

*** ** * ** ***

## Step 3) Update your Azure Data Factory access control

This step is performed by the Azure Data Factory Admin.

To ensure your Azure Data Factory can connect to K, you will need to provide the Azure Data Factory with the correct **Role Assignment**.

* Follow Step 1 to navigate to your Data Factory you wish to profile. You will need to perform the following steps for each Data Factory you wish to profile.

* Open a Data Factory

  ![c94e1c57-2fea-42db-bfc7-52ce0f1ff21c.png](https://docs.kada.ai/__attachments/a_3a4c7a9a4d63c279f09565bc7110a7851f2c0fd37146131dcd916d95c762366d/c94e1c57-2fea-42db-bfc7-52ce0f1ff21c.png?cb=774344b9e838dbbada77cd46cc649b46)  
  ![image-20221206-051956.png](https://docs.kada.ai/__attachments/a_9af49528624a49ff2898fb5d736bac21bf4dcc41d053ec0a91e38d548c963fec/image-20221206-051956.png?cb=b22cc82c2b1cb06a5963629ad6b2f8a3)
* Click on **Access control (IAM)** in the panel and click **Add**

  ![image-20221008-125957.png](https://docs.kada.ai/__attachments/a_291ca0036e97846b56a553fba50ceb4aab28e92c99bb89ce27a3c3f4667d8428/image-20221008-125957.png?cb=c22299722e467236edf737ba6d0f05e4)  
  ![image-20221206-052100.png](https://docs.kada.ai/__attachments/a_1bd20c64965fb4de41b068b946b72d3cdc7944a6db84e3c8ce319ff44a4250ee/image-20221206-052100.png?cb=1ffe048ddc3273e5a6247940f2a9c876)
* Select **Data Factory Contributor**

  ![image-20221008-125552.png](https://docs.kada.ai/__attachments/a_88856a70d20f05224d62be39b4b2c6b082cfc9de86d377dba2b3ea0b5942ccfc/image-20221008-125552.png?cb=d16e8fe863640648a87216d8bf41f894)
* Click **Select Member** . In the side panel add the Service Application you created in Step 2. Click **Select** to add the Service Application.

* Click **Review + Assign** to finish adding the permission.

  ![image-20221206-052302.png](https://docs.kada.ai/__attachments/a_189f63b906fd768a367b1775679413cd019a92753c5f2b01976bc676be1ea230/image-20221206-052302.png?cb=e3ac7769c87c6083b8200a253515567e)

*** ** * ** ***

## Step 4) Add Azure Data Factory as a New Source

This step is performed by the K Admin.

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Click **Add Source** and select **AZURE_DATA_FACTORY**

  ![image-20221008-115610.png](https://docs.kada.ai/__attachments/a_414cf3593d7b89456d3868b7ef164adf92580667e564054ce77274c062f40ef5/image-20221008-115610.png?cb=635b78398e1610982d598e86a72fc5fa)

<!-- -->

* Select **Direct Connect**

* Fill in the **Source Settings** and click **Save \& Next**

  * Name: Give the Azure Data Factory source a name in K. If you have multiple ADFs, each one will need to have a unique name

  * Host: Enter the url e.g. [adf.azure.com](http://adf.azure.com/)

  * **Timeout:** Default is 10, sometimes it may take longer for the API to respond, so we recommend increasing it to 20

  * Update the **Host / Database mapping** (refer to the [mapping documentation](https://docs.kada.ai/k-knowledge-base/host-database-mapping.md)). This step can be completed after the initial load via the guided workflow.

  * Select **Enable Workspace Filtering** if you wish to load only select Workspaces

* Add **Connection Details** and click **Save \& Next**

  * Tenant ID: Add the **Directory (tenant) ID** copied from step 2

  * Client ID: Add the **Application (client) ID** copied from Step 2

  * Client Secret: Add the **Secret ID** copied from Step 2

* **Test your connection** and click **Next**

* If you selected Enabled Workspace Filtering, select the Workspaces you want to load. If you have a lot of workspaces this may take a bit of time to load.

* Click **Finish Setup**

*** ** * ** ***

## Step 4) Schedule Azure Data Factory source load

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Locate your new Azure Data Factory Source and click on the **Schedule Settings** (clock) icon to set the schedule

Note that scheduling a source can take up to 15 minutes to propagate the change.

*** ** * ** ***

## Step 5) Manually run an ad hoc load to test Azure Data Factory

Last updated: July 17, 2026

---
language: "en"
---
# Azure SQL (via Collector method)

This page outlines the Azure SQL Collector versions that are available.

We always recommend using the latest Collector Method to ensure that you can access the latest features.

*** ** * ** ***

## Integration details

|       **Scope**        | **Included** |                        **Comments**                         |
|------------------------|--------------|-------------------------------------------------------------|
| Metadata               | YES          |                                                             |
| Lineage                | YES          | Requires logging to be enabled                              |
| Usage                  | YES          |                                                             |
| Sensitive Data Scanner | NO           | Sensitive data scanner does not currently support Azure SQL |

**Known limitations**

* Queries, macros and procedures must include fully qualified names in order to be correctly parsed. Further improvements are planned to address this limitation.

*** ** * ** ***

## Azure SQL Version History

|                          **Version Number**                          | **Release Month** | **K Platform Compatibility** | **Collector Core Library Compatibility** | **Release changes** |
|----------------------------------------------------------------------|-------------------|------------------------------|------------------------------------------|---------------------|
| [V3.0.0](https://docs.kada.ai/k-knowledge-base/azure-sql-via-collector-method-v3-0-0.md) | Nov 2022          | 5.28+                        | 1.1.1+                                   |                     |

>
Last updated: March 15, 2026

---
language: "en"
---
# Azure SQL (via Collector method) - v3.0.0

## About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

### Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it to pull data for security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pulls the latest metadata

3. Storing and pushing the extracts to your K instance

*** ** * ** ***

## Pre-requisites

**Collector server minimum requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**SQL Server Requirements**

Setting up SQL Server for metadata extraction is a 2 step process.

**Step 1: Establish SQLServer Access**

Apply in MASTER using an Azure SQL Admin user

    CREATE LOGIN kadauser WITH password='PASSWORD';
    CREATE USER kadauser FROM LOGIN kadauser;

Apply per database in scope for metadata collection.

    CREATE USER kadauser FROM LOGIN kadauser;
    GRANT VIEW DEFINITION TO kadauser;
    GRANT VIEW DATABASE STATE to kadauser;
    GRANT CONTROL to kadauser;  -- required for extended events sys.fn_xe_file_target_read_file

The following table should also be available to SELECT by the user created in each database

* INFORMATION_SCHEMA.ROUTINES

* INFORMATION_SCHEMA.VIEWS

* INFORMATION_SCHEMA.TABLE_CONSTRAINTS

* INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE

* INFORMATION_SCHEMA.TABLES

* INFORMATION_SCHEMA.COLUMNS

* sys.foreign_key_columns

* sys.objects

* sys.tables

* sys.schemas

* sys.columns

* sys.databases

**Step 2: Setup Extended Event Logging**

Extended Events Setup is in pilot for Azure SQL

An Azure SQL Admin will need to setup an extended events process to capture Query Execution in SQLServer.

First create or reuse an existing Azure Storage Account. Then create a blob in the example the blob is called `extended-events`

Run the following script to setup Extended Events logging.

Apply per database in scope for metadata collection.
SQL

    CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<REPLACE with your key: abc1234>';

    CREATE DATABASE SCOPED CREDENTIAL [https://your.blob.core.windows.net/extended-events]
    WITH IDENTITY='SHARED ACCESS SIGNATURE',
    SECRET = '< REPLACE WITH YOUR SAS TOKEN: sp=racwdl ...>';

    -- Make sure this file name is unique per database
    CREATE EVENT SESSION [KADA] ON DATABASE
    	ADD EVENT sqlserver.sp_statement_completed (
    		ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_hostname, sqlserver.database_id, sqlserver.database_name, sqlserver.query_hash, sqlserver.session_id, sqlserver.transaction_id, sqlserver.username) WHERE (
    			(
    				[statement] LIKE '%CREATE %'
    				OR [statement] LIKE '%DROP %'
    				OR [statement] LIKE '%MERGE %'
    				OR [statement] LIKE '%FROM %'
    				)
    			AND [sqlserver].[is_system] = (0)
    			AND NOT [statement] LIKE 'Insert into % Values %'
    			AND [sqlserver].[Query_hash] <> (0)
    			)
    		), 
    	ADD EVENT sqlserver.sql_statement_completed (
    	SET collect_statement = (1) ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_hostname, sqlserver.database_id, sqlserver.database_name, sqlserver.query_hash, sqlserver.session_id, sqlserver.transaction_id, sqlserver.username) WHERE (
    		(
    			[statement] LIKE '%CREATE %'
    			OR [statement] LIKE '%DROP %'
    			OR [statement] LIKE '%MERGE %'
    			OR [statement] LIKE '%FROM %'
    			)
    		AND [sqlserver].[is_system] = (0)
    		AND NOT [statement] LIKE 'Insert into % Values %'
    		AND [sqlserver].[Query_hash] <> (0)
    		)
    	) ADD TARGET package0.event_file (SET filename = N'https://your.blob.core.windows.net/extended-events/<REPLACE with your db name: database1>.xel')
    	WITH (MAX_MEMORY = 4096 KB, EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS, MAX_DISPATCH_LATENCY = 30 SECONDS, MAX_EVENT_SIZE = 0 KB, MEMORY_PARTITION_MODE = NONE, TRACK_CAUSALITY = ON, STARTUP_STATE = ON)
    GO

*** ** * ** ***

## Step 1: Create the Source in K

Create a source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File" option**

* Give the source a **Name** - e.g. SQLServer Azure Production

* Add the **Host name** for the SQLServer Azure Instance

* Click **Next** \& **Finish Setup**

*** ** * ** ***

## Step 2: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

1. Go to Platform Settings - Settings. Note down the value of this setting:

   * If using Azure: **storage_azure_storage_account**

   * If using AWS:

     * **storage_root_folder** - the AWS s3 bucket

     * **storage_aws_region** - the region where the AWS s3 bucket is hosted

2. Go to Sources - Edit the Source you have configured. Note down the **landing directory** in the About this Source section.

To connect to the landing directory you will need:

* If using Azure: a **SAS token** to push data to the landing directory. Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

* If using AWS:

  * An **Access key and Secret** . Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

  * OR provide your IAM role to KADA Support to provision access.

*** ** * ** ***

## Step 3: Install the Collector

You can download the Latest Core Library and Azure SQL whl via **Platform Settings → Sources** → **Download Collectors**

Run the following command to install the collector

    pip install kada_collectors_extractors_sqlserver_azure-x.x.x-py3-none-any.whl

You will also need to install the corresponding common library kada_collectors_lib-x.x.x for this collector to function properly.

    pip install kada_collectors_lib-x.x.x-py3-none-any.whl

Note that you will also need an ODBC package installed at the OS level for pyodbc to use as well as a SQLServer ODBC driver, refer to <https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver15>

*** ** * ** ***

## Step 4: Configure the Collector

|  **FIELD**  | **FIELD TYPE** |                   **DESCRIPTION**                   |                                 **EXAMPLE**                                  |
|-------------|----------------|-----------------------------------------------------|------------------------------------------------------------------------------|
| server      | string         | SQLServer Azure server                              | "[mydatabase.database.windows.net](http://mydatabase.database.windows.net/)" |
| host        | string         | The onboarded host value in K                       | "[mydatabase.database.windows.net](http://mydatabase.database.windows.net/)" |
| username    | string         | Username to log into the SQLServer Azure account    | "myuser"                                                                     |
| password    | string         | Password to log into the SQLServer Azure account    |                                                                              |
| databases   | list\<string\> | A list of databases to extract from SQLServer Azure | \["dwh", "adw"\]                                                             |
| driver      | string         | This is the ODBC driver                             | "ODBC Driver 17 for SQL Server"                                              |
| meta_only   | boolean        | Extract metadata only without extended events       | true                                                                         |
| output_path | string         | Absolute path to the output location                | "/tmp/output"                                                                |
| mask        | boolean        | To enable masking or not                            | true                                                                         |
| compress    | boolean        | To gzip the output or not                           | true                                                                         |
| events_name | string         | The created extended event session name             | KADA                                                                         |

**kada_sqlserver_azure_extractor_config.json**
JSON

    {
        "server": "",
        "username": "",
        "password": "",
        "databases": [""],
        "driver": "ODBC Driver 17 for SQL Server",
        "output_path": "/tmp/output",
        "mask": true,
        "compress": true,
        "meta_only": true,
        "host": "",
        "events_name": "KADA"
    }

*** ** * ** ***

## Step 5: Run the Collector

This code sample uses the **kada_sqlserver_azure_extractor.py** for handling the configuration details
Python

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.sqlserver_azure import Extractor

    get_generic_logger('root')

    _type = 'sqlserver_azure'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA SqlServer Azure Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename)
    parser.add_argument('--name', '-n', dest='name', default=_type)
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()
    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

In some scenarios, you may receive an error message about the SSL settings. This error can be resolved via the Open SSL settings. Refer to: <https://github.com/mkleehammer/pyodbc/issues/610#issuecomment-534920201>

    Edited /etc/ssl/openssl.cnf 

    # Change or add

    MinProtocol = TLSv1.0

    CipherString = DEFAULT@SECLEVEL=1

*** ** * ** ***

## Step 6: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

**High Water Mark File**

A high water mark file is created called **sqlserver_azure_hwm.txt**.

Refer to [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md) for more information.

*** ** * ** ***

## Step 7: Push the Extracts to K

Once the files have been validated, you can push the files to the [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md).

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

The following example is how you can orchestrate the Tableau collector using Airflow and push the files to K hosted on Azure. The code is not expected to be used as-is but as a template for your own DAG.
Python

    # built-in
    import os

    # Installed
    from airflow.operators.python_operator import PythonOperator
    from airflow.models.dag import DAG
    from airflow.operators.dummy import DummyOperator
    from airflow.utils.dates import days_ago
    from airflow.utils.task_group import TaskGroup

    from plugins.utils.azure_blob_storage import AzureBlobStorage

    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.tableau import Extractor

    # To be configured by the customer.
    # Note variables may change if using a different object store.
    KADA_SAS_TOKEN = os.getenv("KADA_SAS_TOKEN")
    KADA_CONTAINER = ""
    KADA_STORAGE_ACCOUNT = ""
    KADA_LANDING_PATH = "lz/tableau/landing"
    KADA_EXTRACTOR_CONFIG = {
        "server_address": "http://tabserver",
        "username": "user",
        "password": "password",
        "sites": [],
        "db_host": "tabserver",
        "db_username": "repo_user",
        "db_password": "repo_password",
        "db_port": 8060,
        "db_name": "workgroup",
        "meta_only": False,
        "retries": 5,
        "dry_run": False,
        "output_path": "/set/to/output/path",
        "mask": True,
        "mapping": {}
    }

    # To be implemented by the customer.
    # Upload to your landing zone storage.
    # Change '.csv' to '.csv.gz' if you set compress = true in the config
    def upload():
      output = KADA_EXTRACTOR_CONFIG['output_path']
      for filename in os.listdir(output):
          if filename.endswith('.csv'):
            file_to_upload_path = os.path.join(output, filename)

            AzureBlobStorage.upload_file_sas_token(
                client=KADA_SAS_TOKEN,
                storage_account=KADA_STORAGE_ACCOUNT,
                container=KADA_CONTAINER,
                blob=f'{KADA_LANDING_PATH}/{filename}',
                local_path=file_to_upload_path
            )

    with DAG(dag_id="taskgroup_example", start_date=days_ago(1)) as dag:

        # To be implemented by the customer.
        # Retrieve the timestamp from the prior run
        start_hwm = 'YYYY-MM-DD HH:mm:SS'
        end_hwm = 'YYYY-MM-DD HH:mm:SS' # timestamp now

        ext = Extractor(**KADA_EXTRACTOR_CONFIG)

        start = DummyOperator(task_id="start")

        with TaskGroup("taskgroup_1", tooltip="extract tableau and upload") as extract_upload:
            task_1 = PythonOperator(
                task_id="extract_tableau",
                python_callable=ext.run,
                op_kwargs={"start_hwm": start_hwm, "end_hwm": end_hwm},
                provide_context=True,
            )

            task_2 = PythonOperator(
                task_id="upload_extracts",
                python_callable=upload,
                op_kwargs={},
                provide_context=True,
            )

            # To be implemented by the customer.
            # Timestamp needs to be saved for next run
            task_3 = DummyOperator(task_id='save_hwm')

        end = DummyOperator(task_id='end')

        start >> extract_upload >> end

Last updated: March 21, 2026

---
language: "en"
---
# Azure SQL (via Direct Connect method)

This page will guide you through the setup of Azure SQL in K using the direct connect method.

## Integration details

|       **Scope**        | **Included** |                        **Comments**                         |
|------------------------|--------------|-------------------------------------------------------------|
| Metadata               | YES          |                                                             |
| Lineage                | YES          | Requires logging to be enabled                              |
| Usage                  | YES          |                                                             |
| Sensitive Data Scanner | No           | Sensitive data scanner does not currently support Azure SQL |

**Known limitations**

* Queries, macros and procedures must include fully qualified names in order to be correctly parsed.

*** ** * ** ***

## Step 1) Azure SQL Access

Setting up Azure SQL for metadata extraction is a 2 step process.

**Step 1: Establish SQLServer Access**  
Apply in MASTER using an Azure SQL Admin user replacing kadauser and PASSWORD with your choice of username and password
SQL

    CREATE LOGIN kadauser WITH password='PASSWORD';
    CREATE USER kadauser FROM LOGIN kadauser;

Apply per database in scope for metadata collection.
SQL

    CREATE USER kadauser FROM LOGIN kadauser;
    GRANT VIEW DEFINITION TO kadauser;
    GRANT VIEW DATABASE STATE to kadauser;
    GRANT CONTROL to kadauser;  -- required for extended events sys.fn_xe_file_target_read_file

The following table should also be available to SELECT by the user created in each database

* INFORMATION_SCHEMA.ROUTINES

* INFORMATION_SCHEMA.VIEWS

* INFORMATION_SCHEMA.TABLE_CONSTRAINTS

* INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE

* INFORMATION_SCHEMA.TABLES

* INFORMATION_SCHEMA.COLUMNS

* INFORMATION_SCHEMA.VIEWS

* sys.foreign_key_columns

* sys.objects

* sys.tables

* sys.schemas

* sys.columns

* sys.databases

**Step 2: Setup Extended Event Logging**  
Extended Events Setup is in pilot for Azure SQL.

An Azure SQL Admin will need to setup an extended events process to capture Query Execution in Azure SQL.

Some tuning of the logging parameters may be needed depending on event volumes generated on your Azure SQL instance.

First create or reuse an existing Azure Storage Account.

Then create a blob --- in the example the blob is called `extended-events`.

Run the following script to setup Extended Events logging.  
Apply per database in scope for metadata collection.
SQL

    CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<REPLACE with your key: abc1234>';

    CREATE DATABASE SCOPED CREDENTIAL [https://your.blob.core.windows.net/extended-events]
    WITH IDENTITY='SHARED ACCESS SIGNATURE',
    SECRET = '< REPLACE WITH YOUR SAS TOKEN: sp=racwdl ...>';

    -- Make sure this file name is unique per database: ADD TARGET package0.event_file (SET filename = N'...'
    CREATE EVENT SESSION [KADA] ON DATABASE
    	ADD EVENT sqlserver.sp_statement_completed (
    		ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_hostname, sqlserver.database_id, sqlserver.database_name, sqlserver.query_hash, sqlserver.session_id, sqlserver.transaction_id, sqlserver.username) WHERE (
    			(
    				[statement] LIKE '%CREATE %'
    				OR [statement] LIKE '%DROP %'
    				OR [statement] LIKE '%MERGE %'
    				OR [statement] LIKE '%FROM %'
    				)
    			--AND [sqlserver].[server_principal_name] <> N'USERS_TO_EXCLUDE'
    			AND [sqlserver].[is_system] = (0)
    			AND NOT [statement] LIKE 'Insert into % Values %'
    			AND [sqlserver].[Query_hash] <> (0)
    			)
    		), 
    	ADD EVENT sqlserver.sql_statement_completed (
    	SET collect_statement = (1) ACTION(package0.event_sequence, sqlserver.client_app_name, sqlserver.client_hostname, sqlserver.database_id, sqlserver.database_name, sqlserver.query_hash, sqlserver.session_id, sqlserver.transaction_id, sqlserver.username) WHERE (
    		(
    			[statement] LIKE '%CREATE %'
    			OR [statement] LIKE '%DROP %'
    			OR [statement] LIKE '%MERGE %'
    			OR [statement] LIKE '%FROM %'
    			)
    		--AND [sqlserver].[server_principal_name] <> N'USERS_TO_EXCLUDE'
    		AND [sqlserver].[is_system] = (0)
    		AND NOT [statement] LIKE 'Insert into % Values %'
    		AND [sqlserver].[Query_hash] <> (0)
    		)
    	) ADD TARGET package0.event_file (SET filename = N'https://your.blob.core.windows.net/extended-events/<REPLACE with your db name: database1>.xel')
    	WITH (MAX_MEMORY = 4096 KB, EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS, MAX_DISPATCH_LATENCY = 30 SECONDS, MAX_EVENT_SIZE = 0 KB, MEMORY_PARTITION_MODE = NONE, TRACK_CAUSALITY = ON, STARTUP_STATE = ON)
    GO

*** ** * ** ***

## Step 2) Create the Source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File" option**

* Click **Add Source** and select **Azure SQL**

* Select **Direct Connect** and add your Azure SQL details and click **Next**

* Fill in the **Source Settings** and click **Next**

  * Name: The name you wish to give your Azure SQL Server

  * Host: Add the server location for the Azure SQL Server instance

  * Version number: Set the Azure SQL Server version

  * Extract Meta Only: Set this if extended events is not enabled

* Add the **Connection details** and click **Save \& Next** when connection is successful

  * Host: Add the Azure SQL Server location

  * Username: Add the Azure SQL Server User created in Step 1

  * Password: Add the User password created in Step 1

* **Test your connection** and click **Save**

* Return to the **Sources** page and locate the new **Azure SQL Server** source that you created

* Click on the clock icon to select **Edit Schedule** and set your preferred schedule for the Azure SQL Server load

Note that scheduling a source can take up to 15 minutes to propagate the change.

*** ** * ** ***

## Step 3) Manually run an ad hoc load to test SQL Server setup

Last updated: July 17, 2026

---
language: "en"
---
# Azure Synapse (via Collector method)

This page outlines the Azure Synapse Collector versions that are available.

We always recommend using the latest Collector Method to ensure that you can access the latest features.

*** ** * ** ***

## Integration details

|       **Scope**        | **Included** |                          **Comments**                           |
|------------------------|--------------|-----------------------------------------------------------------|
| Metadata               | YES          | SQL Pools                                                       |
| Lineage                | NO           |                                                                 |
| Usage                  | NO           |                                                                 |
| Sensitive Data Scanner | NO           | Sensitive data scanner does not currently support Azure Synapse |

*** ** * ** ***

## Azure Synapse Version History

|                            **Version Number**                            | **Release Month** | **K Platform Compatibility** | **Collector Core Library Compatibility** | **Release changes** |
|--------------------------------------------------------------------------|-------------------|------------------------------|------------------------------------------|---------------------|
| [V3.0.0](https://docs.kada.ai/k-knowledge-base/azure-synapse-via-collector-method-v3-0-0.md) | Feb 2025          | 5.44+                        | 1.1.1+                                   |                     |

Last updated: March 12, 2026

---
language: "en"
---
# Azure Synapse (via Collector method) - v3.0.0

## About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

### Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it to pull data for security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pulls the latest metadata

3. Storing and pushing the extracts to your K instance

*** ** * ** ***

## Pre-requisites

**Collector server minimum requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**Azure Synapse Requirements**

Setting up Azure Synapse for metadata extraction using a service principal (Application).

**Step 1: Registering an Entra App for KADA Azure Synapse Collector Application**

Create an Entra Application for the kada synapse collector or reusing an existing kada application in Entra.

Generate a secret for the Entra Application and note down the application id, tenant id.

**Step 2: Establish Azure Synapse Access**

Apply in MASTER using an Azure Synapse Admin user

    CREATE USER [<ENTRA APPLICATION NAME>] FROM EXTERNAL PROVIDER; 

Apply per database in scope for metadata collection.

    CREATE USER [<ENTRA APPLICATION NAME>] FROM EXTERNAL PROVIDER;
    GRANT VIEW DEFINITION TO [<ENTRA APPLICATION NAME>];
    GRANT VIEW DATABASE STATE TO [<ENTRA APPLICATION NAME>];

The following table should also be available to SELECT by the user created in each database

* INFORMATION_SCHEMA.ROUTINES

* INFORMATION_SCHEMA.VIEWS

* INFORMATION_SCHEMA.TABLE_CONSTRAINTS

* INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE

* INFORMATION_SCHEMA.TABLES

* INFORMATION_SCHEMA.COLUMNS

* sys.foreign_key_columns

* sys.objects

* sys.tables

* sys.schemas

* sys.columns

* sys.databases

Synapse has the concept of a serverless and dedicated sql pool per workspace.

If you use both serverless and dedicated sql pools in a workspace you will need to onboard each as a separate source in K.

For serverless sql pools the master database can't be selected for metadata extraction.

To onboard multiple Synapse workspace each workspace will need to be onboarded as a new source in K.

*** ** * ** ***

## Step 1: Create the Source in K

Create a source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File" option**

* Give the source a **Name** - e.g. SQLServer Azure Production

* Add the **Host name** for the SQLServer Azure Instance

* Click **Next** \& **Finish Setup**

*** ** * ** ***

## Step 2: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

1. Go to Platform Settings - Settings. Note down the value of this setting:

   * If using Azure: **storage_azure_storage_account**

   * If using AWS:

     * **storage_root_folder** - the AWS s3 bucket

     * **storage_aws_region** - the region where the AWS s3 bucket is hosted

2. Go to Sources - Edit the Source you have configured. Note down the **landing directory** in the About this Source section.

To connect to the landing directory you will need:

* If using Azure: a **SAS token** to push data to the landing directory. Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

* If using AWS:

  * An **Access key and Secret** . Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

  * OR provide your IAM role to KADA Support to provision access.

*** ** * ** ***

## Step 3: Install the Collector

You can download the Latest Core Library and Azure Synapse whl via **Platform Settings → Sources** → **Download Collectors**

Run the following command to install the collector

    pip install kada_collectors_extractors_azure_synapse-3.0.0-py3-none-any.whl

You will also need to install the corresponding common library kada_collectors_lib-x.x.x for this collector to function properly.

    pip install kada_collectors_lib-x.x.x-py3-none-any.whl

Note that you will also need an ODBC package installed at the OS level for pyodbc to use as well as a SQLServer ODBC driver, refer to <https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver15>

*** ** * ** ***

## Step 4: Configure the Collector

|     **FIELD**      | **FIELD TYPE** |                          **DESCRIPTION**                          |                  **EXAMPLE**                   |
|--------------------|----------------|-------------------------------------------------------------------|------------------------------------------------|
| client             | string         | Onboarded client in Azure to access Azure Synapse                 |                                                |
| secret             | string         | Onboarded client secret in Azure to access Azure Synapse          |                                                |
| tenant             | string         | Tenant ID of where Azure Synapse exists                           |                                                |
| server             | string         | Azure Synapse server                                              | "\<workspace-name\>.sql.azuresynapse.net,1433" |
| host               | string         | The onboarded host value in K                                     | "\<workspace-name\>.sql.azuresynapse.net,1433" |
| database_name      | string         | The name of the database that will be used to test the connection | master                                         |
| databases          | list\<string\> | A list of databases to extract from SQLServer Azure               | \["dwh", "adw"\]                               |
| driver             | string         | This is the ODBC driver                                           | "ODBC Driver 17 for SQL Server"                |
| meta_only          | boolean        | Extract metadata only                                             | true                                           |
| output_path        | string         | Absolute path to the output location                              | "/tmp/output"                                  |
| mask               | boolean        | To enable masking or not                                          | true                                           |
| compress           | boolean        | To gzip the output or not                                         | true                                           |
| connection_timeout | integer        | Timeout in seconds for Synapse Sql Pool connection                | 30                                             |

**kada_azure_synapse_extractor_config.json**
JSON

    {
        "client": "",
        "secret": "",
        "tenant": "",
        "server": "",
        "host": "",
        "driver": "ODBC Driver 17 for SQL Server",
        "database_name": "master",
        "databases": [""],
        "output_path": "/tmp/output",
        "mask": true,
        "compress": true,
        "meta_only": true,
        "connection_timeout": 30
    }

*** ** * ** ***

## Step 5: Run the Collector

This code sample uses the **kada_azure_synapse_extractor.py** for handling the configuration details
Python

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.azure_synapse import Extractor

    get_generic_logger('root')

    _type = 'azure_synapse'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA Azure Synapse Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename)
    parser.add_argument('--name', '-n', dest='name', default=_type)
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()
    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

In some scenarios, you may receive an error message about the SSL settings.

This error can be resolved via the Open SSL settings. Refer to: <https://github.com/mkleehammer/pyodbc/issues/610#issuecomment-534920201>

    Edited /etc/ssl/openssl.cnf 

    # Change or add

    MinProtocol = TLSv1.0

    CipherString = DEFAULT@SECLEVEL=1

*** ** * ** ***

## Step 6: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

**High Water Mark File**

A high water mark file is created called **azure_synapse_hwm.txt**.

Refer to [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md) for more information.

*** ** * ** ***

## Step 7: Push the Extracts to K

Once the files have been validated, you can push the files to the [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md).

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

The following example is how you can orchestrate the Tableau collector using Airflow and push the files to K hosted on Azure. The code is not expected to be used as-is but as a template for your own DAG.
Python

    # built-in
    import os

    # Installed
    from airflow.operators.python_operator import PythonOperator
    from airflow.models.dag import DAG
    from airflow.operators.dummy import DummyOperator
    from airflow.utils.dates import days_ago
    from airflow.utils.task_group import TaskGroup

    from plugins.utils.azure_blob_storage import AzureBlobStorage

    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.tableau import Extractor

    # To be configured by the customer.
    # Note variables may change if using a different object store.
    KADA_SAS_TOKEN = os.getenv("KADA_SAS_TOKEN")
    KADA_CONTAINER = ""
    KADA_STORAGE_ACCOUNT = ""
    KADA_LANDING_PATH = "lz/tableau/landing"
    KADA_EXTRACTOR_CONFIG = {
        "server_address": "http://tabserver",
        "username": "user",
        "password": "password",
        "sites": [],
        "db_host": "tabserver",
        "db_username": "repo_user",
        "db_password": "repo_password",
        "db_port": 8060,
        "db_name": "workgroup",
        "meta_only": False,
        "retries": 5,
        "dry_run": False,
        "output_path": "/set/to/output/path",
        "mask": True,
        "mapping": {}
    }

    # To be implemented by the customer.
    # Upload to your landing zone storage.
    # Change '.csv' to '.csv.gz' if you set compress = true in the config
    def upload():
      output = KADA_EXTRACTOR_CONFIG['output_path']
      for filename in os.listdir(output):
          if filename.endswith('.csv'):
            file_to_upload_path = os.path.join(output, filename)

            AzureBlobStorage.upload_file_sas_token(
                client=KADA_SAS_TOKEN,
                storage_account=KADA_STORAGE_ACCOUNT,
                container=KADA_CONTAINER,
                blob=f'{KADA_LANDING_PATH}/{filename}',
                local_path=file_to_upload_path
            )

    with DAG(dag_id="taskgroup_example", start_date=days_ago(1)) as dag:

        # To be implemented by the customer.
        # Retrieve the timestamp from the prior run
        start_hwm = 'YYYY-MM-DD HH:mm:SS'
        end_hwm = 'YYYY-MM-DD HH:mm:SS' # timestamp now

        ext = Extractor(**KADA_EXTRACTOR_CONFIG)

        start = DummyOperator(task_id="start")

        with TaskGroup("taskgroup_1", tooltip="extract tableau and upload") as extract_upload:
            task_1 = PythonOperator(
                task_id="extract_tableau",
                python_callable=ext.run,
                op_kwargs={"start_hwm": start_hwm, "end_hwm": end_hwm},
                provide_context=True,
            )

            task_2 = PythonOperator(
                task_id="upload_extracts",
                python_callable=upload,
                op_kwargs={},
                provide_context=True,
            )

            # To be implemented by the customer.
            # Timestamp needs to be saved for next run
            task_3 = DummyOperator(task_id='save_hwm')

        end = DummyOperator(task_id='end')

        start >> extract_upload >> end

Last updated: March 21, 2026

---
language: "en"
---
# Batch Jobs

Updated in Version 6.0  

|-------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Job**                             | **Description**                                                                                                                                                                                                |
| **APPLY COLLECTION INSTANCE RULES** | Applies rule‑based linking as defined in the Linking Rules under the Collection settings. Run this job to update all collection links after a linking rule change.                                             |
| **CROSS SOURCE LINKING**            | Matches and links related assets across multiple source systems. Creates unified entity views by resolving duplicates, aligning identifiers, and establishing cross‑system relationships.                      |
| **DAILY**                           | Executes routine daily processing tasks such as lineage changes, usage statistics and non real time index updates                                                                                              |
| **DATABASE OPTIMISATION**           | Performs automated database tuning tasks including statistics updates, index maintenance, and storage cleanup. Improves query performance and overall platform stability.                                      |
| **GATHER METRICS AND STATS**        | Updates platform statistics such as usage and load frequency                                                                                                                                                   |
| **HIGHLY FREQUENT**                 | Runs high‑frequency micro‑jobs that process incremental updates or event‑driven changes such as new feed updates.                                                                                              |
| **INDEX REBUILD**                   | Rebuilds search index after significant changes such as upgrades or major platform maintenance                                                                                                                 |
| **INDEX UPDATE**                    | Updates the search index for any non-real time changes.                                                                                                                                                        |
| **INTRA DAY**                       | Executes multiple times throughout the day to refresh asset details                                                                                                                                            |
| **OBJECT COLLAPSE**                 | Consolidates duplicate and redundant assets                                                                                                                                                                    |
| **OBJECT PURGE**                    | Removes expired, obsolete, or superseded assets based on the retention period set                                                                                                                              |
| **SCORE UPDATE OR REFRESH**         | Recalculates or refreshes platform scores such as trust score.                                                                                                                                                 |
| **SOURCE LOAD**                     | Load metadata and log data from sources into the platform. Handles file processing, schema validation, and log loads into K. After a source is load, the Daily job is required to see the results in platform. |
| **SUMMARY TABLE REFRESH**           | Regenerates summary or aggregate tables used for platform stats.                                                                                                                                               |
| **SYNC SOLR**                       | Synchronises platform data with a Solr search index. Updates indexed documents to maintain search accuracy and relevance.                                                                                      |
| **WATCHER**                         | Monitors data pipelines, system health, or specific datasets for anomalies or threshold breaches. Triggers alerts or follow‑up actions when conditions are met.                                                |
| **WEEKLY**                          | Runs weekly scheduled tasks such as full reconciliations, data clean up and platform maintenance.                                                                                                              |

Last updated: July 26, 2026

---
language: "en"
---
# BigQuery (via Collector method)

This page outlines the BigQuery Collector versions that are available.

We always recommend using the latest Collector Method to ensure that you can access the latest features.

## Integration details

|       **Scope**        | **Included** |                      **Comments**                       |
|------------------------|--------------|---------------------------------------------------------|
| Metadata               | YES          |                                                         |
| Lineage                | YES          |                                                         |
| Usage                  | YES          |                                                         |
| Sensitive Data Scanner | No           | Sensitive Scanner does not currently support Big Query. |

*** ** * ** ***

## BigQuery Version History

|                          **Version Number**                          | **Release Month** | **K Platform Compatibility** | **Collector Core Library Compatibility** |  **Release changes**   |
|----------------------------------------------------------------------|-------------------|------------------------------|------------------------------------------|------------------------|
| [V 3.0.0](https://docs.kada.ai/k-knowledge-base/bigquery-via-collector-method-v3-0-0.md) | Nov 2022          | 5.23 - 5.25                  | 1.0.1                                    | First version released |

Last updated: March 15, 2026

---
language: "en"
---
# BigQuery (via Collector method) - v3.0.0

## About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

### Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it to pull data for security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pulls the latest metadata

3. Storing and pushing the extracts to your K instance

*** ** * ** ***

## Pre-requisites

**Collector Server Minimum Requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**BigQuery Requirements**

* Access to BigQuery

*** ** * ** ***

## Step 1: Establish BigQuery Access

This step is performed by the Google Cloud Admin

* Create a Service Account by going to the Google Cloud Admin or clicking on this [link](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts/create)

  * Give the Service Account a name (e.g. KADA BQ Integration)

  * Select the Projects that include the BigQuery instance(s) that you want to catalog

  * Click Save

* Create a Service Token

  * Click on the Service Account

  * Select the Keys tab. Click on Create new key

  * Select the JSON option. After clicking 'CREATE', the JSON file will automatically download to your device.

* Add permission grants on the Service Account by going to IAM page

  * Click on ADD

  * Add the Service Account to the 'New principals' field.

  * Grant the following roles this principal:

    * BigQuery Job User

    * BigQuery Metadata Viewer

    * BigQuery Read Session User

    * BigQuery Resource Viewer

  * Click SAVE

*** ** * ** ***

## Step 2: Create the Source in K

Create a BigQuery source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File system" option**

* Give the source a **Name** - e.g. BigQuery Production

* Add the **Host name** for the BigQuery Server

* Click **Finish Setup**

*** ** * ** ***

## Step 3: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

1. Go to Platform Settings - Settings. Note down the value of this setting:

   * If using Azure: **storage_azure_storage_account**

   * If using AWS:

     * **storage_root_folder** - the AWS s3 bucket

     * **storage_aws_region** - the region where the AWS s3 bucket is hosted

2. Go to Sources - Edit the Source you have configured. Note down the **landing directory** in the About this Source section.

To connect to the landing directory you will need:

* If using Azure: a **SAS token** to push data to the landing directory. Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

* If using AWS:

  * An **Access key and Secret** . Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

  * OR provide your IAM role to KADA Support to provision access.

*** ** * ** ***

## Step 4: Install the Collector

You can download the Latest Core Library and whl via **Platform Settings → Sources** → **Download Collectors**

Run the following command to install the collector

    pip install kada_collectors_extractors_<version>-none-any.whl

You will also need to install the common library kada_collectors_lib for this collector to function properly.

    pip install kada_collectors_lib-<version>-none-any.whl

Under the covers this uses the BigQuery Client API and may have OS dependencies see <https://cloud.google.com/bigquery/docs/reference/libraries>

*** ** * ** ***

## Step 5: Configure the Collector

|    **FIELD**     | **FIELD TYPE** |                       **DESCRIPTION**                       |                      **EXAMPLE**                      |
|------------------|----------------|-------------------------------------------------------------|-------------------------------------------------------|
| regions          | list\<string\> | List of valid regions to inspect                            | "us"                                                  |
| projects         | list\<string\> | List of project ids to inspect across the regions specified | "kada-data"                                           |
| host             | string         | This is the host that was onboarded in K for BigQuery       | "bigquery"                                            |
| json_credentials | JSON           | Service account credentials JSON                            | {"type": "service_account", "project_id": "...", ...} |
| output_path      | string         | Absolute path to the output location                        | "/tmp/output"                                         |
| mask             | boolean        | To enable masking or not                                    | true                                                  |
| compress         | boolean        | To gzip the output or not                                   | true                                                  |

**kada_bigquery_extractor_config.json**

    {
        "regions": [],
        "projects": [],
        "host": "",
        "json_credentials": {},
        "output_path": "/tmp/output",
        "mask": true,
        "compress": true
    }

*** ** * ** ***

## Step 6: Run the Collector

This is the wrapper script: **kada_bigquery_extractor.py**

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.bigquery import Extractor

    get_generic_logger('root')

    _type = 'bigquery'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA BigQuery Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename)
    parser.add_argument('--name', '-n', dest='name', default=_type)
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()
    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

*** ** * ** ***

## Step 7: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

**High Water Mark File**

A high water mark file is created called **bigquery_hwm.txt**.

Refer to [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md) for more information.

*** ** * ** ***

## Step 8: Push the Extracts to K

Once the files have been validated, you can push the files to the [K landing directory.](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

The following example is how you can orchestrate the Tableau collector using Airflow and push the files to K hosted on Azure. The code is not expected to be used as-is but as a template for your own DAG.
Python

    # built-in
    import os

    # Installed
    from airflow.operators.python_operator import PythonOperator
    from airflow.models.dag import DAG
    from airflow.operators.dummy import DummyOperator
    from airflow.utils.dates import days_ago
    from airflow.utils.task_group import TaskGroup

    from plugins.utils.azure_blob_storage import AzureBlobStorage

    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.tableau import Extractor

    # To be configured by the customer.
    # Note variables may change if using a different object store.
    KADA_SAS_TOKEN = os.getenv("KADA_SAS_TOKEN")
    KADA_CONTAINER = ""
    KADA_STORAGE_ACCOUNT = ""
    KADA_LANDING_PATH = "lz/tableau/landing"
    KADA_EXTRACTOR_CONFIG = {
        "server_address": "http://tabserver",
        "username": "user",
        "password": "password",
        "sites": [],
        "db_host": "tabserver",
        "db_username": "repo_user",
        "db_password": "repo_password",
        "db_port": 8060,
        "db_name": "workgroup",
        "meta_only": False,
        "retries": 5,
        "dry_run": False,
        "output_path": "/set/to/output/path",
        "mask": True,
        "mapping": {}
    }

    # To be implemented by the customer.
    # Upload to your landing zone storage.
    # Change '.csv' to '.csv.gz' if you set compress = true in the config
    def upload():
      output = KADA_EXTRACTOR_CONFIG['output_path']
      for filename in os.listdir(output):
          if filename.endswith('.csv'):
            file_to_upload_path = os.path.join(output, filename)

            AzureBlobStorage.upload_file_sas_token(
                client=KADA_SAS_TOKEN,
                storage_account=KADA_STORAGE_ACCOUNT,
                container=KADA_CONTAINER,
                blob=f'{KADA_LANDING_PATH}/{filename}',
                local_path=file_to_upload_path
            )

    with DAG(dag_id="taskgroup_example", start_date=days_ago(1)) as dag:

        # To be implemented by the customer.
        # Retrieve the timestamp from the prior run
        start_hwm = 'YYYY-MM-DD HH:mm:SS'
        end_hwm = 'YYYY-MM-DD HH:mm:SS' # timestamp now

        ext = Extractor(**KADA_EXTRACTOR_CONFIG)

        start = DummyOperator(task_id="start")

        with TaskGroup("taskgroup_1", tooltip="extract tableau and upload") as extract_upload:
            task_1 = PythonOperator(
                task_id="extract_tableau",
                python_callable=ext.run,
                op_kwargs={"start_hwm": start_hwm, "end_hwm": end_hwm},
                provide_context=True,
            )

            task_2 = PythonOperator(
                task_id="upload_extracts",
                python_callable=upload,
                op_kwargs={},
                provide_context=True,
            )

            # To be implemented by the customer.
            # Timestamp needs to be saved for next run
            task_3 = DummyOperator(task_id='save_hwm')

        end = DummyOperator(task_id='end')

        start >> extract_upload >> end

Last updated: March 21, 2026

---
language: "en"
---
# BigQuery (via Direct Connect method)

This page will walk through the setup of BigQuery in K using the direct connect method.

## Integration details

|       **Scope**        | **Included** |                      **Comments**                       |
|------------------------|--------------|---------------------------------------------------------|
| Metadata               | YES          |                                                         |
| Lineage                | YES          |                                                         |
| Usage                  | YES          |                                                         |
| Sensitive Data Scanner | NO           | Sensitive Scanner does not currently support Big Query. |

*** ** * ** ***

## Step 1) Setup a Google Cloud Service Account

This step is performed by the Google Cloud Admin.

* Create a Service Account by going to the [Google Cloud Admin](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts/create)

  * Give the Service Account a name (e.g. KADA BQ Integration)

  * Select the Projects that include the BigQuery instance(s) that you want to catalog

  * Click Save

* Create a Service Token

  * Click on the Service Account

    ![image-20220922-043354.png](/__attachments/a_5f143a25a045603daf774345f73b4fefc674f4609b37efccb02297e3c46db2bb/image-20220922-043354.png?cb=042b4db6535ccc7b425c06f426fb1c4a)
  * Select the **Keys** tab. Click on **Create new key**

    ![image-20220922-043443.png](/__attachments/a_45a4b069ecb11d18506aee7dff3af7e4479eb4575e7f0c003401c92438b3d5dc/image-20220922-043443.png?cb=ea5a69036eb73a2ad28fd62da78e8260)
  * Select the **JSON** option. After clicking 'CREATE', the JSON file will automatically download to your device. Provide this to the user(s) that will complete the next steps.

    ![image-20220920-130318.png](/__attachments/a_9b3fdcd6f2d5403b09dc053bfc71ac920a30a118f7ba3754e321c1749c6c3c9b/image-20220920-130318.png?cb=2d6530fc12f15b2bebd91e2303c41454)
* Add grants on the Service Account by going to the [IAM page](https://console.cloud.google.com/iam-admin/)

  * Click on **ADD**

    ![image-20220922-043501.png](/__attachments/a_42737fe1f1bcf7f94955242898e0c483a4a7714cb32ab6a8c7e4f259c91ec29c/image-20220922-043501.png?cb=d89c5b0f3070048a470133ad763f5c92)
  * Add the Service Account to the 'New principals' field.

    ![image-20221022-123622.png](/__attachments/a_b5b6f0a30817a84eb96b2068de135e35ba3c30ab114ca667149c2118778d422c/image-20221022-123622.png?cb=2bef2d65bea73eaecb0d3b4c43455747)
  * Grant the following roles to this principal:

    * BigQuery Job User

    * BigQuery Metadata Viewer

    * BigQuery Read Session User

    * BigQuery Resource Viewer

      ![image-20220922-044346.png](/__attachments/a_90ae3a8562fbda817f952dd968825b84ea23ba22d0946cc44886bbecc52e118a/image-20220922-044346.png?cb=32152b7e84db4cb57363566238db2145)
  * Click **SAVE**

*** ** * ** ***

## Step 2) Connecting K to BigQuery

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Click **Add Source** and select **BigQuery**

  ![image-20260715-125736.png](https://docs.kada.ai/__attachments/a_7047dbec50f468af4f53244fc37e15ff545208827037e1062f3a83698ab2e791/image-20260715-125736.png?cb=6b3ca8596b1f7ab37a66f37778b8852d)

* Select **Direct Connect**

* Fill in the **Source Settings** and click **Save \& Next**

  * Name: The name you wish to give your BigQuery Service in K

  * Host: Add your BigQuery Host Name (e.g. [cloud.google.com](http://cloud.google.com/))

  * Region: Select the region your Service is located in (check with your admin if you are unsure)

* Add the **Connection details** and click **Save \& Next** when connection is successful

  * Credentials: Copy the content of the Credentials.json created in Step 1

* **Test your connection** and click **Save**

* **Select the Databases** you wish to load into K and click **Finish Setup**

  * All databases will be listed. If you have a lot of databases this may take a few seconds to load.

* Return to the **Sources** page and locate the new **BigQuery** source that you loaded

* Click on the clock icon to select **Edit Schedule** and set your preferred schedule for the BigQuery load

Note that scheduling a source can take up to 15 minutes to propagate the change.

*** ** * ** ***

## Step 3) Manually run an ad hoc load to test BigQuery setup

Last updated: July 17, 2026

---
language: "en"
---
# Bulk Action

UPDATED IN VERSION 6.1

Through **Bulk Action** you can:

* Update properties for multiple data assets

* Easily link multiple data assets to a list

* Easily link multiple data assets to a warning notice

Once a bulk action has been created, you can choose to save it and set a schedule to have it run on a regular basis.

* For example: Set a bulk action to run monthly to automatically assign a specific data owner to newly created data assets for a domain.

Bulk Action is only available to Data Governance, Data Manager, and K Admin users, except for **Add to List**, which is available to all users

*** ** * ** ***

## Accessing Bulk Action

You can access Bulk Action in three ways:

**Option 1) Search Page** --- run a search and select assets from the results, then open Bulk Action from the toolbar.  
![image-20260531-105440.png](https://docs.kada.ai/__attachments/a_28c278c14fcebf86b57b5263bcea0e093420923bf5ceca448995a05fd89d4695/image-20260531-105440.png?cb=d5a0ee252af22d66da503bec844208fa)

**Option 2) Via Saved Search**--- On the search page save your filter settings and then open Bulk Action from the Saved Search page.  
![image-20260805-114648.png](https://docs.kada.ai/__attachments/a_2c2618c4dfc8c15bed4abd423e2d622a317319f32e68b6e1601c3c66f97f6c05/image-20260805-114648.png?cb=6ebd85f77df33c81cf9f9a0696c6663e)

**Option 3) Via Lists** --- open a list and select assets, then open Bulk Action from the list toolbar.  
![image-20260531-105726.png](https://docs.kada.ai/__attachments/a_080897646450b300dba76043a8b9f3d1064e7aa391a01f2e410f901a95d5ddb8/image-20260531-105726.png?cb=40208208a86e2739703991e7cfd84b26)

*** ** * ** ***

## How to complete a Bulk Action

* **Step 1)** Open Bulk Action for the assets you want to update via Search, Filters, or Lists.

* **Step 2)** Choose the **attribute** you want to update. You can select multiple attributes to update at the same time.

  ![image-20260531-105836.png](https://docs.kada.ai/__attachments/a_98bfe570a5de30cc8597cec77201dafb4419a6b464ebc98031cf67f57ac972cd/image-20260531-105836.png?cb=7b9349b7b63d616b408d79eb6e79e49c)
* **Step 3)** Confirm the **Scope** of the bulk update. If multiple types of data assets have been selected, you can restrict the update to a specific asset type (e.g. Tables and Columns only).

* **Step 4)** Confirm the type of **Action**. There are up to 4 action types:

  * **Clear** --- delete all current values stored

  * **Remove only** --- delete a specific value (e.g. remove one of several Stewards)

  * **Replace** --- replace the current values with a new one

  * **Add to existing** --- add additional values alongside existing ones

* **Step 5)** Review the Action Settings --- confirm the correct target has been selected, then choose to:

  * Run now

  * Save for future use

  * Schedule for a future run

* **Step 6)** After clicking Save or Run, you will be taken to the **Actions Centre** where you can view saved and previously run actions.

*** ** * ** ***

## About the Actions Centre

Access the Actions Centre from the side menu. It shows all saved actions and run history.

For any saved action, the menu lets you:

* Review the target list of data assets the action updates

* View the run history for that action

* Edit the action, including changing its run schedule

* Run the action immediately

* Delete the action

![image-20260531-110236.png](https://docs.kada.ai/__attachments/a_6cd7b2a166ffe485ec10ef33400547a3b5c0170f1ad75a2d5092b26d6bae42d1/image-20260531-110236.png?cb=2ef62686557b87ddcfd953208ee6a8fa)

*** ** * ** ***

Change history  
**Version 6.1** · 2026-08-17

* UPDATED Add to List bulk action now available to all users

Last updated: August 15, 2026

---
language: "en"
---
# Bulk Edit Collections

Updated in Version 6.0

Data Governance Managers, Data Managers, and K Admin Users have the ability to Bulk Edit Collections.

This page will walk you through how to perform a bulk edit for collections and key things to look out for.

*** ** * ** ***

## Accessing Bulk Edit for Collections

The Bulk Edit function for collections is accessed via **Platform Settings → Collections**.

Where you see the **Generate Bulk Edit** or **Import Bulk Edit** icon you will have the ability to perform a bulk update function.  
![image-20260314-101035.png](https://docs.kada.ai/__attachments/a_9ae95784beaab6a1bc322512fe5a0faf5f79f1a803e57243fba70837d69ef25b/image-20260314-101035.png?cb=506adb1be7335ea2f793ed56d3dbe093)

You will be able to perform a Bulk Edit for all Glossary, Data Management and Data Governance collections.

For Platform defined K Collections, Bulk Edit will only be available for some collections.

*** ** * ** ***

## How to Perform a Bulk Edit for Collections

As each Collection can have a unique structure and field settings, you will need to download a Collection-specific Excel template. **Do not** re-purpose a collection template.

1. Click on the **Generate Bulk Edit** file icon for the specific Collection that you would like to update

2. Access your Collection template in your **downloads** folder

3. Review the **Guide** tab to understand the Excel layout and how to use the template. Key tips include:

   * Always use the green coloured **Collection Instance** tab to make your updates

   * The first row will highlight which columns are Mandatory vs. Optional

   * The next two rows will provide field property information and tips --- check the tips to ensure you have a successful upload

   * Where a field is a look-up field that requires an existing list of options (e.g. Data Owner), you will need to first ensure the Data Owner collection is updated with all the options you require. You will not be able to select/add a Data Owner that has not been pre-created in the K platform

4. After you have updated the Excel file, click **Import Bulk Edit**

   * Check that the Collection selected is the one that you intend to update

     ![image-20240510-115813.png](/__attachments/a_2dd2c33c46acc1c6ad8eb31536ea66cfdd2eba1b25cfd4c7047f5f3ab30f2f88/image-20240510-115813.png?cb=def609e813a79e6bac5a3ce9407e2a10)
5. Confirm Import Type and click **Next**

   * **Update only:** K Platform will modify the existing collection details based on the Excel template

   * **Delete and reload:** K will delete all collection details including existing linkages to the collection, and the collection will be created from scratch. Data assets will need to be re-linked to the new collection instance

6. Attach your updated Excel and click **Import**

7. You will be taken to **Jobs**where you can monitor the progress of your Bulk Edit import

   ![image-20260314-101410.png](https://docs.kada.ai/__attachments/a_89f191aadd06c129cc85f8a5295312a9dfdf5fd0a0bbbf1ad59e59ff8480391a/image-20260314-101410.png?cb=de95bdc61d1a0cb730a982ae6a1cdd61)

**Key checks to prevent upload fails**

* Where the property is a lookup value (e.g. Data Owner, User Name), ensure there are no typos and that all values specified are valid options. Any typos or invalid values will result in the entire file being rejected during the import process.

* Check that all blank cells have been intentionally left blank. Blank cells will overwrite any existing data in K and delete any existing information.

* Keep an original version of the downloaded template on hand as a back-up to reverse any errors made during a bulk upload process.

*** ** * ** ***

## Reviewing historical Bulk Edit Imports

You can view all historical Bulk Edit Imports in the **Import** tab.

It's a great way to check the historical import status and understand who performed previous bulk edits.  
![image-20260314-101548.png](https://docs.kada.ai/__attachments/a_5b1eda54e0787c34e0259bd4f2269a21ce086944a0b7e1bd4a7aed408f60a1d5/image-20260314-101548.png?cb=8fe12f84d448e65178a7c6da189ea76a)

*** ** * ** ***

## Bulk edit collection limitations

Uploading collection data via this feature unfortunately has some limitations

* Changes are not recorded in change history

* Created by is not set to the user that uploaded the file

* Alias is not available in the Bulk edit template

* Alternate name is not available in the Bulk edit template

Last updated: July 26, 2026

---
language: "en"
---
# Bulk Edit via Excel Upload

Updated in Version 6.0

Bulk Edit via Excel Upload lets you export a template of your data assets, make changes in Excel, and import the file back into K. This is a preferred approach when:

* Changes need to be socialised, reviewed, or approved with stakeholders outside of K

* The required changes (e.g. descriptions) are already documented in a spreadsheet

**Access:** This function is currently limited to Table, Column, Report, and Sheet objects and is only available to K Administrators.

**Note:** Changes made via Bulk Edit via Excel are not logged in the asset's change log.

*** ** * ** ***

## Step 1) Select assets and generate your template

You can select assets to add to your Bulk Edit Excel template via three options:

* **Option 1) Search Results** --- run a search and select assets from the results

  ![image-20260531-111245.png](https://docs.kada.ai/__attachments/a_1fac237ed0a50d01a50976dbfc677d2989d36c7e56f8deacdf9ce2b585c7bc9e/image-20260531-111245.png?cb=febf00212c6451e69f38dedf44a411ea)
* **Option 2) Filters** --- apply filters to scope your asset list

  ![image-20260531-111330.png](https://docs.kada.ai/__attachments/a_22470430deb8b79e837e2c8415fdc53f38c21154ac17f85421dc9068fc2cf380/image-20260531-111330.png?cb=f7b2cae372179b32016d6e68ce278434)
* **Option 3) Lists** --- select assets from within a list

  ![image-20260531-111510.png](https://docs.kada.ai/__attachments/a_c817a8ea7eae49dc3752be044341bb4ec00c31646b62901bb1ffd7d5f6c1e5e1/image-20260531-111510.png?cb=da98a978359f11f66ae8b9caf02fefe9)

By default, the following properties are included in the template:

* Business name, Description, Tags, Classification, Domains, Verified for, Do not use for, Owner, Stewards

  ![image-20260531-111630.png](https://docs.kada.ai/__attachments/a_38c5c9ac149ad8d6ff0d6511a1b16d795496025b6c725c70f5e213000a965bb3/image-20260531-111630.png?cb=ed46ac483f8d1c781046a56a713d6ae3)

You can also add up to 6 additional properties (15 properties total can be updated at once). Review the template properties and click **Create**. You will be taken to the My Application Results page where you can download the file when it is ready.

*** ** * ** ***

## Step 2) Edit the Excel template

* Open the file in Excel and follow the instructions on the first (Instructions) tab.

* Edit the properties for your assets and save the file locally.

**Key things to note:**

* Cells highlighted in **grey** are locked to protect the workbook structure --- do not edit these.

* When re-imported, **all metadata properties will be replaced** --- blank cells will overwrite existing data in K.

* **Orange columns** have restricted values (e.g. only pre-defined Domains can be entered). Refer to the corresponding orange tab for the list of valid values. If a value is missing, create it in K first, then re-generate the template.

* Enter multiple values (e.g. multiple Domains) by separating each with a comma.

**Always check for and delete blank rows.** Previously edited rows that have been cleared are treated as "activated" --- K will attempt to upload them and the job will fail. To delete a blank row:

* Right-click the tab and click **Unprotect Sheet** (no password required)

* Right-click the blank row and click **Delete**

*** ** * ** ***

## Step 3) Upload your edited file

* Go to the **Import Bulk Edit File** page via K Applications.

  ![image-20260531-112014.png](https://docs.kada.ai/__attachments/a_13b9c7ddd3307d113e5095fe2780c47987f9ff9c6762f3423c38d7c8a09ca7ee/image-20260531-112014.png?cb=7a3e9775f4708d302f08fb6f697f3ef6)
* Click **Upload Bulk Edit File** , select your edited file, and click **Import**.

  ![image-20260531-112109.png](https://docs.kada.ai/__attachments/a_ead8cc31487ac6e942d55e29ca9af33276d3464aef00df93c979b5a439b7897b/image-20260531-112109.png?cb=577fdd6877332ecda0c8064e3d7f9575)
* If the file uploads without errors, it will show a successful upload. If there are issues, it will fail and provide error details.

  ![image-20260531-112206.png](https://docs.kada.ai/__attachments/a_941c5b78b966a4353071f2bf97914598040b10e2eaaae5b82f4d8df2b49a389b/image-20260531-112206.png?cb=b997abb16da2fab25894cf885bad0cc5)

**Note on error rows:** The row number in the error log is incremental across all tabs. For example, if the Tables tab has 500 records and the Reports tab has 100 records, an error on row 550 refers to row 50 in the Reports tab.

Last updated: July 26, 2026

---
language: "en"
---
# Bulk Functions

Updated in Version 6.0

K's Bulk Functions speed up the process of updating data properties, linking assets to collections, adding tags, and creating lists --- allowing governance teams to make large-scale changes efficiently rather than asset by asset.

*** ** * ** ***

## What You Can Do in Bulk

|       **Function**       |                                                **Description**                                                 |
|--------------------------|----------------------------------------------------------------------------------------------------------------|
| Update Properties        | Edit data asset properties (e.g. owner, steward, description, classification) across multiple assets at once.  |
| Add / Remove Tags        | Apply or remove governance tags across a selected set of assets in a single action.                            |
| Link to Collections      | Associate multiple assets with one or more collections or domains simultaneously.                              |
| Create Lists             | Build curated lists of assets for governance workflows, reporting, or team reference.                          |
| Assign Owners / Stewards | Assign or reassign data ownership across multiple assets at once --- useful during governance uplift programs. |

*** ** * ** ***

## How to Use Bulk Functions

1. **Search, filter** or open a **list**to identify the assets you want to update

2. **Select** one or more assets from the search results

3. **Choose a Bulk Function** from the actions menu

4. **Apply** the change --- K will process the update across all selected assets

*** ** * ** ***

## Related

* [Automate Data Tagging](https://docs.kada.ai/k-knowledge-base/automate-data-tagging.md)

* [Data Owner \& Steward Dashboard](https://docs.kada.ai/k-knowledge-base/data-owner-steward-dashboards.md)

Last updated: July 26, 2026

---
language: "en"
---
# Bulk Lineage Mapper

Updated in Version 6.0

There are scenarios where K may not automatically create lineage links. For example:

* An upstream data source has been loaded as a Manual Source in K

* Data loaded into K has insufficient metadata to enable automated lineage

Where you know that tables, columns, dataset tables, or dataset fields are connected, you can use the **Lineage Mapper** application to bulk-create lineage links using matching rules.

*** ** * ** ***

## Accessing Lineage Mapper

Click on **K Applications** in the side menu, then select **Lineage Mapper**.  
![image-20260531-112435.png](https://docs.kada.ai/__attachments/a_ea80884797ef2daf5929053034bb0627473814446e5fd491e186280d00691a0c/image-20260531-112435.png?cb=f3a7f9de3cc2fb24e45be7c3cdde6047)

*** ** * ** ***

## How to bulk map lineage

* **Step 1)** Confirm the asset type combination and source you want to link. You can link:

  * Tables / Dataset Tables → Tables / Dataset Tables

  * Columns / Dataset Fields → Columns / Dataset Fields

  If linking at the Table level, you can also choose the lineage depth (e.g. whether to also link underlying columns).  
  ![image-20260531-112526.png](https://docs.kada.ai/__attachments/a_912e5e3bb9772a4b2e9d7f06a8c709c554b1555a84e907baca835ce23bfa36e5/image-20260531-112526.png?cb=67b87fde0f43aebbf01b16add8862db4)

<!-- -->

* **Step 2)** Confirm the table, column, or dataset that is to be linked.

* **Step 3)** Confirm how names should be matched. You can specify that the column or table name follows one of these formats:

  * Name

  * \<prefix\> + Name

  * Name + \<suffix\>

  * \<prefix\> + Name + \<suffix\>

* **Step 4)** Review the examples shown --- the first three asset names are surfaced to help you verify the name formatting is correct.

* **Step 5)** Click **Confirm** . You will be taken to **My Applications → Jobs → Lineage Mapping** to monitor the job.

  * While the job is running, click the **Cross** icon to cancel it.

  * Once completed, the mapping cannot be reversed --- any errors will need to be corrected manually.

    ![image-20260531-112611.png](/__attachments/a_f37160474288d35d72940c1cd90e4cb8355a4d95318586dba3db27c63a8b1afe/image-20260531-112611.png?cb=ba523048969cf360f20cabf935123fe4)

<!-- -->

* Step 6) You will be taken to the **Jobs Page→ Lineage Mapping**page where you can see the progress of your Lineage Mapping Job.

  * While the job is running, you can click the **Cross** icon to cancel the job

  * After the job has been completed, you will not be able to 'reverse' the mapping job. Any changes or errors will need to be manually edited one at a time.

  ![image-20260531-112807.png](https://docs.kada.ai/__attachments/a_7b484915317dee64fcf7a260695ec994b80a31911a80423e33c914833d757fd2/image-20260531-112807.png?cb=0503d76f44c0691cd69c4cf082f0dc77)

*** ** * ** ***

## What does a manual lineage link look like?

Manual lineage links are displayed with a **dashed line** and an **(M)** symbol, making them easy to distinguish from automatically detected lineage.  
![image-20260531-112845.png](https://docs.kada.ai/__attachments/a_034d46e772760880e03ddaf2b110fc445cea14837c78d56b3128da2d25256db6/image-20260531-112845.png?cb=a4e12f67b637f8a71aef9dbf6177cc20)

Last updated: July 26, 2026

---
language: "en"
---
# Bulk set-up of users and teams

This page will walk through the process of setting up user profiles in bulk via a file upload.

This process is currently under review for improvements (moving to a UI based upload / bulk action). Your feedback is greatly appreciated.

*** ** * ** ***

## Upload File Format

Ensure the file you create follows the following format  

|      **Property**      |                                                                **Value**                                                                 |
|------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| Encoding               | UTF-8 (No BOM)                                                                                                                           |
| File Delimiter         | \| *(pipe)*                                                                                                                              |
| Headers                | Ordered and present as per contract all in uppercase.                                                                                    |
| Record Quoting         | All fields enclosed in double quotes. "value". Any double quotes inside the field value must be escaped with an additional double quote. |
| Record Delimiter       | \\                                                                                                                                       |
| *(new line character)* |
| Empty Fields           | Non mandatory fields may be left empty. Note that an empty field must still be doubled quoted.                                           |

*** ** * ** ***

## Step 1) Generate user upload file

* Create a csv file with the following parameters:

  * Extract contains a row per user.

  * File name: **USERS_YYYYMMDDHHMMSS.csv**

  * Daily Load: Extracts new users.

  * Historical Load: Full snapshot of users.

  * Follows the file format above.

|   **Column**   | **Data Type** |   **Value Mandatory**    |                                                                                 **Description**                                                                                 |
|----------------|---------------|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| USERNAME       | STRING        | Y                        | Unique id for the user. Typically email of the user especially if SSO with Active Directory is used.                                                                            |
| FIRST_NAME     | STRING        | Y if IS_LOGIN_USER = 'Y' | Example: Jane                                                                                                                                                                   |
| LAST_NAME      | STRING        | Y if IS_LOGIN_USER = 'Y' | Example: Doe                                                                                                                                                                    |
| EMAIL          | STRING        | Y if IS_LOGIN_USER = 'Y' | Example: [Jane.Doe@email.com](mailto:Jane.Doe@email.com)                                                                                                                        |
| DESCRIPTION    | STRING        | N                        | Description or Title of the user                                                                                                                                                |
| IS_SYSTEM_USER | STRING        | N                        | Is the user a human user or a system account? One of: (Y, N)                                                                                                                    |
| IS_LOGIN_USER  | STRING        | N                        | Will the user log into KADA? One of: (Y, N). Default to 'N' if not provided                                                                                                     |
| ROLES          | STRING        | N                        | Comma separated string of roles. Defaults to "kada_user" if no role is provided. Valid values: kada_user, kada_admin, kada_data_manager, kada_business_user, kada_data_gov_user |
| GROUP_NAME     | STRING        | N                        | The group (Team) the user belongs to. Must match a value from GROUP MAPPING interface                                                                                           |
| USER_ID        | STRING        | N                        | INTERNAL use only. Leave empty                                                                                                                                                  |

*** ** * ** ***

## Step 2) Generate Team upload file

* Create a csv file using AD / LDAP with the following parameters:

  * Extract contains a row per group.

  * File name: **GROUPS_YYYYMMDDHHMMSS.csv**

  * Daily Load: Extracts new groups.

  * Historical Load: Full snapshot of groups.

In this context **Groups = Teams**  

| **Column**  | **Data Type** | **Value Mandatory** |                          **Description**                          |
|-------------|---------------|---------------------|-------------------------------------------------------------------|
| NAME        | STRING        | Y                   | Name of group (i.e. Team). Must be unique.                        |
| PARENT_NAME | STRING        | N                   | Name of the parent team. Must match the team name in another row  |
| DESCRIPTION | STRING        | N                   | Description of the team for presentation within the K application |

*** ** * ** ***

## Step 3) Upload user and group files to K

Requires K administrator access and access to your K instance storage container.

1. Log into K

2. Go to Platform Settings → Sources

3. Click **Add Source**.

   1. Select MICROSOFT_WINDOW_AD or LDAP.

   2. Select Load from File

   3. Add a Name: e.g. Active Directory

   4. Add a Host: e.g. ActiveDirectory

   5. Note down the landing folder e.g. windows_ad

   6. Click Next and Finalise the setup

4. Access the folder created via Azure Storage Explorer and the SAS token provided by K (for SaaS) or the storage container for your Cloud implementation

5. Upload the files above to the landing directory for the source you created e.g. kada-data/lz/windows_ad/landing

6. Go back to the Sources page. Click run manual load action on the source you created

7. Check the monitor page for the status of the job. On completion, your new teams and users will have been created

Last updated: March 13, 2026

---
language: "en"
---
# Business Lineage

NEW IN RELEASE 6.1

## What is Business Lineage?

Business Lineage lets people understand an asset's upstream or downstream context --- **without having to open and read a full technical lineage graph**.

It provides a simplified view vs. technical lineage, reducing the need to expand lineage maps to answer questions such as where does the data come from.  
![image-20260805-235636.png](https://docs.kada.ai/__attachments/a_c4fe9b2c71520d859bdbfd18d5b9f2b36cd2f1d640b066f87c2328d0c90f31f9/image-20260805-235636.png?cb=9e2cefaf96fdf75f4c0858c0f73645f7)  
![image-20260805-235058.png](https://docs.kada.ai/__attachments/a_3da748592b95449f7387af89d7604748ca50ad8e225714a04bd2c0e647719602/image-20260805-235058.png?cb=6a45e4c8d95ff9aa73a68e9660129d3b)

*** ** * ** ***

## How it works

Once Business Lineage is configured, all assets assigned a Layer will be:

* **Connected** to its upstream and downstream assets across the Raw, Consumption, and Presentation layers --- any intermediate assets, such as staging or temp tables, are hidden from this view.

* **Rolled up**, for Terms, Warnings, Issues, and DQ Tests linked at column level, to the parent table (or from Sheets/Dataset Fields to their parent Report/Dataset Table).

* **Pushed down**, for its Layer position and System of Record, to its own columns.

* **Propagated**, for Terms, Warnings, Issues, DQ Tests, and Source in K, along technical lineage to every downstream asset.

System of Record is the one property that does both: it pushes down to its own table's columns and separately propagates along lineage to every downstream table.

Links created by business lineage can be **suppressed** if they're noise or incorrect --- suppressing an item only affects that one relationship on that one asset; it doesn't stop the item from continuing downstream. See [Suppressing and Restoring Propagated Items](https://docs.kada.ai/k-knowledge-base/suppressing-and-restoring-propagated-items.md) for the one action that does block downstream propagation (suppressing the lineage edge itself, not the item).

Visit [Business Lineage Rules](https://docs.kada.ai/k-knowledge-base/business-lineage-rules.md)for more detailed information and examples on how it works in practice

*** ** * ** ***

## Why it matters

|                    Outcome                     |                                                                            What changes for users                                                                             |
|------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Faster trust decisions                         | A user opening a report or table sees upstream definitions, warnings, quality failures, and open issues directly on the asset --- no need to trace the lineage graph by hand. |
| Governance configured once, applied everywhere | Governance teams set up System of Record and layers (Raw / Consumption / Presentation) a single time; the daily job keeps every downstream asset up to date automatically.    |
| Noise stays controllable                       | Anything propagated is clearly distinguished from directly-linked metadata, and can be suppressed by users with edit permissions such as Data Owners and Stewards.            |

*** ** * ** ***

## Key Business Lineage Concepts

|                         Term                         |                                                                                                                                                                                                   Meaning                                                                                                                                                                                                   |
|------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Layer**                                            | A table-level property: None, Raw, Consumption, or Presentation. Layer determines whether an asset's Business Lineage tab is shown, and what shows up in the layered upstream view. Adding a Layer to a Table level object will propagate the layer value to its children (e.g. Columns) without you having to perform the action.                                                                          |
| **System of Record**                                 | A Collection marking where data originates from. Linked to table-level assets in the Raw layer only. It is automatically pushed down to that table's own columns and propagates downstream to Consumption and Presentation assets.                                                                                                                                                                          |
| **Source in K**                                      | An auto-derived link showing where an asset's data is sourced from within K. This is useful to quickly identify which upstream source (e.g. Database) a Report is connected to without having to look at technical lineage.                                                                                                                                                                                 |
| **Roll up**                                          | Terms, Warnings, Issues, and DQ Tests linked at column level are automatically added to the parent table. Roll-up is one level only --- a rolled-up item is *display only* and does not itself continue propagating onward from the parent.                                                                                                                                                                 |
| **Push down**                                        | Layer and System of Record travel from a table down to its own columns only.                                                                                                                                                                                                                                                                                                                                |
| **Propagated from upstream**                         | An item that reached an asset by travelling technical lineage (not roll-up or push-down) from an upstream asset. Propagated items are tagged with a "business_" prefix on their relationship name (e.g. `relates_to` becomes `business_relates_to`) --- that prefix is your visual cue that the item arrived on its own rather than being linked directly.                                                  |
| **Suppression**                                      | Hiding a specific propagated relationship on a specific asset, without deleting the underlying link. It only affects that one relationship, on that one asset, and doesn't stop the item from reaching assets further downstream.                                                                                                                                                                           |
| **Business Lineage edge vs. Technical Lineage edge** | Both can be added manually between two tables, but they're not interchangeable. A manually-added Business Lineage edge only establishes the upstream Layer reference --- it does not carry Terms, Warnings, or other attributes. A manually-added Technical Lineage edge behaves like a real, discovered lineage edge and carries everything, the same as an edge Business Lineage relies on automatically. |

*** ** * ** ***

## What stops propagation

Three things behave differently to what you might expect. Worth knowing before you start troubleshooting a "missing" item:

* **Deleting an asset in the path** stops everything beyond it from showing that chain's upstream context --- not the layer reference, not any attribute travelling through it.

* **Deleting an attribute itself** (a Term, Issue, DQ test, Source in K reference, or System of Record object) also stops it from propagating further, **except Warnings** --- a deleted or inactive Warning keeps propagating as if nothing happened.

* **Hidden flags don't block anything.** `hidden_in_K` and `hidden_in_source` are visibility settings, not deletions, and have no effect on propagation.

Suppressing an item is also easy to mix up with suppressing the connection carrying it: suppressing an item (a Term, Warning, Issue, or DQ test) is local to that one asset and doesn't interrupt what's downstream. Suppressing or deactivating the lineage edge itself is the stronger action --- it blocks everything from crossing that edge. See [Suppressing and Restoring Propagated Items](https://docs.kada.ai/k-knowledge-base/suppressing-and-restoring-propagated-items.md).

*** ** * ** ***

## Who does what

|                                                                          Guide                                                                           |                    Who it's for                    |                                                          What it covers                                                           |
|----------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
| [Setting Up Business Lineage](https://docs.kada.ai/k-knowledge-base/setting-up-business-lineage.md)                                                                          | Data Governance, Data Manager, Admin               | Enabling Business Lineage, configuring System of Record, assigning Layers (individually and in bulk)                              |
| [Viewing Business Lineage on an Asset](https://docs.kada.ai/k-knowledge-base/understanding-business-lineage-viewing-lineage-terms-warnings-issues-and-data-quality-tests.md) | All users                                          | Where propagated Terms, Warnings, Issues, and DQ Tests show up across the Overview, Quality, and Lineage tabs, and in the sidebar |
| [Suppressing and Restoring Propagated Items](https://docs.kada.ai/k-knowledge-base/suppressing-and-restoring-propagated-items.md)                                            | Users with Edit access, Data Owners, Data Stewards | Hiding a propagated item that's noise, and restoring it later, plus how that differs from suppressing a lineage edge              |
| Business Lineage Extract                                                                                                                                 | All users                                          | Getting upstream context through Ask K, and exporting business lineage data                                                       |

*** ** * ** ***

## Permissions Summary

|                                                                      Action                                                                      |                                                 Who                                                  |
|--------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|
| View business lineage on a profile (roll-ups, push-downs, Terms, Warnings, Issues, Upstream DQ tests, Lineage tab, map, Layer, System of Record) | All K roles                                                                                          |
| Configure Business Lineage Settings page                                                                                                         | Data Governance, Data Manager, or Admin                                                              |
| Edit Layer / System of Record, incl. bulk actions                                                                                                | Data Governance, Data Manager, or Admin                                                              |
| Link, suppress, or restore Terms / Warnings / Issues / Upstream DQ tests                                                                         | Anyone granted permission via the Role Permission page, **or** the asset's Data Owner / Data Steward |
| Run the Business Lineage extract                                                                                                                 | All K roles                                                                                          |

*** ** * ** ***

## Current Limitations

* Business Lineage is calculated daily (part of the daily job schedule).

* There's no audit log of suppression actions in this release.

* Business Lineage itself is **disabled by default** at the platform level and must be turned on in Settings.

*** ** * ** ***

Change history  
**Version 6.1** · 2026-08-17

* NEW New page. Business Lineage feature dashboards released as part of Version 6.1

Last updated: August 15, 2026

---
language: "en"
---
# Business Lineage Rules

NEW IN RELEASE 6.1

Business Lineage propagation follows a small set of rules. This page groups all 12 rules into five types, each with a diagram you can read on its own, plus the exact mechanics underneath if you want them.  
**Tip:**How to quickly identify if an attribute has been propagated or directly linked

* Propagated attributes always start with the prefix "business_". For example:

  * Direct link: relates_to

  * Propagated link: business_relates_to

*** ** * ** ***

## The Three Propagation Mechanisms

Almost every rule below is a variation of one of these three ways an attribute can move from one asset to another.  

|   Propagation Mechanisms   |                           Direction                           |                            Applies to                            |
|----------------------------|---------------------------------------------------------------|------------------------------------------------------------------|
| 1. Roll up                 | Column → its own parent table (one level only)                | Terms, Warnings, Issues, DQ Tests                                |
| 2. Push down               | Table → its own columns (one level only)                      | Layer, System of Record                                          |
| 3. Propagate along lineage | Any upstream asset → any downstream asset, any number of hops | Terms, Warnings, Issues, DQ Tests, Source in K, System of Record |

*** ** * ** ***

## 5 Groups of Propagation Rules

* **Core Mechanics** (rules 1-4) --- How Layer, System of Record, and other attributes move by default

* **How Paths Combine** (rules 5-6) --- What you see when table-level and column-level paths both reach the same asset

* **Manual Edges** (rule 7) --- Two ways to manually connect two tables, and what each one carries

* **Deletion** (rules 8-9) --- Removing an asset or attribute stops propagation, with one exception

* **Hidden Flags \& Suppression** (rules 10-12) --- What looks like it should block propagation, and what actually does

*** ** * ** ***

## Group 1: Core Mechanics

How Layer, System of Record, and other attributes move by default.  
![image-20260806-130135.png](https://docs.kada.ai/__attachments/a_5ff828af8a05900e60093136de1e8b3b705a45707c553106c2c8513340de2515/image-20260806-130135.png?cb=bcdbe3d8afc74cec58a545ab5694fbec)

### Rule 1: Layer Is Inherited by Columns, Not Chosen by Them

If a table is set to the Raw layer, every column in it automatically shows Raw too, greyed out, because you can't change it at the column level. Set the table to Consumption instead, and its columns follow.
Under the hood  
* Follow the chain from Raw to Consumption to Presentation.

* Each table's own columns show the same Layer as their table, marked as inherited and locked --- you can't override it at the column level.

* This is push-down in action: Layer is only ever set on the table itself, and it flows down exactly one level, to that table's own columns.

### Rule 2: Column Attributes Roll Up Once --- Then Stop

Link a Term to Column A and it correctly shows up on Column B too, thanks to a direct lineage link between them. It also shows up on both columns' parent tables (rolled up). But it doesn't keep travelling further down the table chain.
Under the hood  
* Column A is linked directly to a Term.

* Because Column A feeds Column B through lineage, Column B picks up that Term too, marked as propagated rather than direct.

* Both parent tables, Raw and Consumption, then show the Term as rolled up.

* It stops there: Consumption 2 and Presentation show no Term at all --- a roll-up doesn't keep travelling downstream on its own.

### Rule 3: System of Record Propagates Across Tables and Pushes Down

Set a System of Record on the Raw table, and it shows up on Consumption and Consumption 2 downstream (arriving "via propagated upstream"), and it also correctly shows up on all of Raw's own columns.
Under the hood  
* The System of Record is linked directly to the Raw table.

* From there it travels two ways at once:

  * Downstream to Consumption and Consumption 2, where each shows it as arrived from upstream.

  * Down to Raw's own columns, where each shows it as inherited from its table.

* System of Record is the only attribute that does both at the same time --- but it never moves directly from one column to another the way Terms do.

### Rule 4: Table-Level Attributes Never Push Down to Columns

A Warning linked to the Raw table correctly shows up on Consumption and Presentation downstream. But check the columns that belong to those tables, the Warning isn't there. Warnings roll up and propagate along lineage, but are never pushed down.
Under the hood  
* A Warning is linked directly to the Raw table.

* It travels downstream as expected: Consumption and Presentation both pick it up, marked as propagated.

* But look at the columns belonging to those same tables --- none of them show the Warning.

* This is the mirror image of Rule 3: Layer and System of Record travel down to columns, but Warnings, Terms, Issues, and DQ Tests never do.

*** ** * ** ***

## Group 2: How Paths Combine

What you see when table-level and column-level paths both reach the same asset.  
![image-20260806-130047.png](https://docs.kada.ai/__attachments/a_846998214c7314510b9e038f40f6bcafbe74daf3718541e3ecc453287d00a90c/image-20260806-130047.png?cb=600d76ce48b83f531cf9178134452f4d)

### Rule 5: Linked at Both Levels? Both Views Show It --- Independently

If WarningA is linked directly to the Raw table, and separately, directly, to Column A within that table (Column A belongs to Raw), you'll see it in both places when you look: once on the table-level Business Lineage view, once on the column-level Business Lineage view. Neither link depends on the other.
Under the hood  
* The Warning is linked directly in two separate places:

  * Once to the Raw table itself.

  * Once to Column A, which belongs to that table.

* Each link works completely on its own:

  * The table-level link carries the Warning table-to-table as normal.

  * The column-level link separately rolls it up and carries it column-to-column.

* Changing or suppressing one link has no effect on the other.

### Rule 6: How Layer and Attribute Paths Combine Downstream

When you land on an asset's Business Lineage tab, what you see is a blend: the Layer badge always comes from the table-level chain only, while any Warnings, Terms, or Issues you see could have arrived via the table chain, the column chain, or both.
Under the hood  
* Table A is set to the Raw layer and linked to a Warning.

* Table B is set to Consumption, and its Column 123 (which inherits Consumption) is separately linked to that same Warning.

* On Table B's Business Lineage tab:

  * The Layer badge shows Raw --- coming only from the table-to-table path.

  * The Upstream Warnings list shows the Warning from both the table path and the column path combined, since Warnings can arrive either way.

*** ** * ** ***

## Group 3: Manual Edges

Two ways to manually connect two tables --- and they carry different things.  
![image-20260806-130210.png](https://docs.kada.ai/__attachments/a_f1dee88b14b1acff7dde4d87a0cbe9de3180b2f6036db954a4eab9b8850bc31c/image-20260806-130210.png?cb=609bdb6ef9a2ecd17ec7ed1e82c71a10)

### Rule 7: Manually-Added Edges: Business vs. Technical Lineage

There are two different ways to manually draw a connection between two tables, and they're not interchangeable. A manually-added Business Lineage edge only tells K "this table sits upstream for layer purposes," it won't carry Terms or other attributes across. A manually-added Technical Lineage edge behaves like a real, discovered lineage edge: it carries everything.
Under the hood  
* With a manually-added **Business Lineage** edge: Table A (which has a Term linked to it) is manually connected to Table B, which then connects on to Table C.

  * Table B correctly picks up Table A as its Raw layer source.

  * But the Term never reaches Table B.

* With a manually-added **Technical Lineage** edge instead:

  * The same kind of connection now carries the Term as expected.

  * Table A is correctly recognized as the Raw source two hops away, at Table C.

*** ** * ** ***

## Group 4: Deletion

Removing an asset or attribute stops propagation --- with one exception.  
![image-20260806-130015.png](https://docs.kada.ai/__attachments/a_4741c140dc3665ac3c814247dde04c21e1d0e2ea6f486f9d8d8a69e2f6814b20/image-20260806-130015.png?cb=ed6c3aba6d2050dd2e4e289206cb5b90)

### Rule 8: A Deleted Asset in the Path Stops Propagation

If a table partway along a lineage chain gets deleted, nothing beyond it will show that chain's upstream context anymore, not the layer reference, not any attribute that was travelling through it.
Under the hood  
* Table X feeds into Table A, which feeds into Table B, and a Term is linked to Table A.

* Once Table A is deleted, Table B stops showing both the upstream Raw reference and the Term --- neither one makes it across the gap left by the deleted table.

* This holds whether the deleted asset is a table or a column.

### Rule 9: A Deleted Attribute Stops Propagating --- Except Warnings

Deleting the Term, Issue, DQ test, Source in K reference, or System of Record object itself (as opposed to the table it's attached to) also stops it from propagating. Warnings are the one exception, a deleted or inactive Warning keeps propagating as if nothing happened.
Under the hood  
* **General case:** a Source in K reference gets deleted while the table it's linked to, Table A, stays intact. Table B, downstream, no longer shows that Source in K reference.

* **Exception:** do the same thing with a Warning instead --- delete the Warning but keep Table A, and Table B still shows the Warning anyway. Warnings ignore this rule entirely.

*** ** * ** ***

## Group 5: Hidden Flags \& Suppression

What looks like it should block propagation, and what actually does.  
![image-20260806-125935.png](https://docs.kada.ai/__attachments/a_5a19649d7e5ff83b5cd99a966217142c821fc1c63e4346b939a378e164c62fb4/image-20260806-125935.png?cb=a688dd620d4d85db4af83c12bebd827e)

### Rule 10: "Hidden" Flags Do Not Block Propagation

Some objects can be marked hidden, either hidden within K itself, or hidden at the source system. That's a visibility setting, not a deletion, and it doesn't interrupt propagation.
Under the hood  
* Mark a Source in K reference as hidden in K, then link it to Table A, which feeds into Table B.

* Table B still correctly shows both the upstream Raw reference and the Source in K reference --- the hidden flag made no difference.

* The same is true if it's marked hidden at the source system instead.

* Only deletion blocks propagation --- hiding something never does.

### Rule 11: Suppressing an Item Doesn't Stop It Propagating --- Unless Deleted Too

If you suppress a Term on one table because it's noise there, that suppression is local, the term still shows up correctly on the next table downstream, exactly as if you hadn't touched it. Suppression only disappears from the chain if the table carrying it is also deleted.
Under the hood  
* Table X feeds into Table A, which feeds into Table B.

* Suppress a Term on Table A without deleting anything, and Table B still shows that Term arriving from upstream --- suppressing it on Table A didn't stop it travelling further.

* Now also delete Table A, and Table B stops showing it entirely.

* Deletion overrides everything; suppressing an item by itself does not.

### Rule 12: Suppressing the Lineage Edge Itself Does Block Propagation

There's a difference between suppressing an attribute on an asset and suppressing or deactivating the lineage connection between two assets. The first doesn't block downstream propagation (see Rule 11). The second does, because the path itself is broken.
Under the hood  
* Table X feeds into Table A, but this time the lineage connection between them is marked suppressed or inactive --- not the tables or the Term themselves.

* Table B no longer shows the upstream Raw reference or the Term, even though nothing was deleted.

* A suppressed connection simply isn't a path K will carry anything across.

*** ** * ** ***

Change history  
**Version 6.1** · 2026-08-17

* New New page. Business Lineage feature released as part of Version 6.1.

Last updated: August 15, 2026

---
language: "en"
---
# Bytehouse (via Collector method)

This page outlines the ByteHouse Collector versions that are available.

We always recommend using the latest Collector Method to ensure that you can access the latest features.

*** ** * ** ***

## Integration details

|       **Scope**        | **Included** |                  **Comments**                   |
|------------------------|--------------|-------------------------------------------------|
| Metadata               | YES          |                                                 |
| Lineage                | YES          |                                                 |
| Usage                  | NO           | Not currently available from Bytehouse - due Q4 |
| Sensitive Data Scanner | N/A          |                                                 |

*** ** * ** ***

## ByteHouse Version History

|                          **Version Number**                          | **Release Month** | **K Platform Compatibility** | **Collector Core Library Compatibility** |  **Release changes**   |
|----------------------------------------------------------------------|-------------------|------------------------------|------------------------------------------|------------------------|
| [V3.0.0](https://docs.kada.ai/k-knowledge-base/bytehouse-via-collector-method-v3-0-0.md) | Sept 2024         | 5.40 - 5.42                  | 1.1.5 - 1.1.9                            | First version released |

Last updated: March 12, 2026

---
language: "en"
---
# Bytehouse (via Collector method) - v3.0.0

## About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

### Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it to pull data for security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pulls the latest metadata

3. Storing and pushing the extracts to your K instance

*** ** * ** ***

## Pre-requisites

**Collector Server Minimum Requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**ByteHouse Requirements**

* Access to the following tables

  1. `system.databases`

  2. `system.tables`

  3. `system.columns`

*** ** * ** ***

## Step 1: Enabling logging

Ensure logging has been enabled in Bytehouse.

*** ** * ** ***

## Step 2: Create the Source in K

Create a ByteHouse source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File system" option**

* Give the source a **Name** - e.g. ByteHouse Production

* Add the **Host name** for the ByteHouse Server

* Click **Finish Setup**

*** ** * ** ***

## Step 3: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

1. Go to Platform Settings - Settings. Note down the value of this setting:

   * If using Azure: **storage_azure_storage_account**

   * If using AWS:

     * **storage_root_folder** - the AWS s3 bucket

     * **storage_aws_region** - the region where the AWS s3 bucket is hosted

2. Go to Sources - Edit the Source you have configured. Note down the **landing directory** in the About this Source section.

To connect to the landing directory you will need:

* If using Azure: a **SAS token** to push data to the landing directory. Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

* If using AWS:

  * An **Access key and Secret** . Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

  * OR provide your IAM role to KADA Support to provision access.

*** ** * ** ***

## Step 4: Install the Collector

You can download the latest Core Library and whl via **Platform Settings → Sources** → **Download Collectors**

Run the following command to install the collector

    pip install kada_collectors_extractors_<version>-none-any.whl

You will also need to install the common library kada_collectors_lib for this collector to function properly.

    pip install kada_collectors_lib-<version>-none-any.whl

*** ** * ** ***

## Step 5: Configure the Collector

The ByteHouse collector only extracts metadata and does not extract or process query usage on the database.  

|     **FIELD**     | **FIELD TYPE** |                                                              **DESCRIPTION**                                                              |                 **EXAMPLE**                  |
|-------------------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------|
| api_key           | string         | The API Key for ByteHouse, you can generate one via the Console                                                                           | "xasdaxcv"                                   |
| server            | string         | ByteHouse gateway, these are regionally specific - see <https://docs.byteplus.com/en/docs/bytehouse/docs-supported-regions-and-providers> | "gateway.aws-ap-southeast-1.bytehouse.cloud" |
| port              | integer        | The port to connect to the ByteHouse instance, generally this is 19000                                                                    | 19000                                        |
| host              | string         | The onboarded host in K for the ByteHouse Source                                                                                          | "gateway.aws-ap-southeast-1.bytehouse.cloud" |
| tenant_account_id | string         | This value can be found in the ByteHouse console under the Tenant Management Tab and Basic Information                                    | "123456778"                                  |
| meta_only         | boolean        | Currently we only support meta only as true                                                                                               | true                                         |
| output_path       | string         | Absolute path to the output location                                                                                                      | "/tmp/output"                                |
| mask              | boolean        | To enable masking or not                                                                                                                  | true                                         |
| compress          | boolean        | To enable compression or not to .csv.gz                                                                                                   | true                                         |
| timeout           | integer        | Timeout setting in seconds                                                                                                                | 80000                                        |

**kada_bytehouse_extractor_config.json**

    {
        "api_key": "",
        "server": "",
        "port": 19000,
        "tenant_account_id": "",
        "host": "",
        "output_path": "/tmp/output",
        "mask": true,
        "compress": true,
        "meta_only": true,
        "timeout": 80000
    }

*** ** * ** ***

## Step 6: Run the Collector

This is the wrapper script: **kada_bytehouse_extractor.py**

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.bytehouse import Extractor

    get_generic_logger('root')

    _type = 'bytehouse'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA Bytehouse Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename)
    parser.add_argument('--name', '-n', dest='name', default=_type)
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()
    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

*** ** * ** ***

## Step 7: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

**High Water Mark File**

A high water mark file is created called **bytehouse_hwm.txt**.

Refer to [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md) for more information.

*** ** * ** ***

## Step 8: Push the Extracts to K

Once the files have been validated, you can push the files to the [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md).

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

The following example is how you can orchestrate the Tableau collector using Airflow and push the files to K hosted on Azure. The code is not expected to be used as-is but as a template for your own DAG.
Python

    # built-in
    import os

    # Installed
    from airflow.operators.python_operator import PythonOperator
    from airflow.models.dag import DAG
    from airflow.operators.dummy import DummyOperator
    from airflow.utils.dates import days_ago
    from airflow.utils.task_group import TaskGroup

    from plugins.utils.azure_blob_storage import AzureBlobStorage

    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.tableau import Extractor

    # To be configured by the customer.
    # Note variables may change if using a different object store.
    KADA_SAS_TOKEN = os.getenv("KADA_SAS_TOKEN")
    KADA_CONTAINER = ""
    KADA_STORAGE_ACCOUNT = ""
    KADA_LANDING_PATH = "lz/tableau/landing"
    KADA_EXTRACTOR_CONFIG = {
        "server_address": "http://tabserver",
        "username": "user",
        "password": "password",
        "sites": [],
        "db_host": "tabserver",
        "db_username": "repo_user",
        "db_password": "repo_password",
        "db_port": 8060,
        "db_name": "workgroup",
        "meta_only": False,
        "retries": 5,
        "dry_run": False,
        "output_path": "/set/to/output/path",
        "mask": True,
        "mapping": {}
    }

    # To be implemented by the customer.
    # Upload to your landing zone storage.
    # Change '.csv' to '.csv.gz' if you set compress = true in the config
    def upload():
      output = KADA_EXTRACTOR_CONFIG['output_path']
      for filename in os.listdir(output):
          if filename.endswith('.csv'):
            file_to_upload_path = os.path.join(output, filename)

            AzureBlobStorage.upload_file_sas_token(
                client=KADA_SAS_TOKEN,
                storage_account=KADA_STORAGE_ACCOUNT,
                container=KADA_CONTAINER,
                blob=f'{KADA_LANDING_PATH}/{filename}',
                local_path=file_to_upload_path
            )

    with DAG(dag_id="taskgroup_example", start_date=days_ago(1)) as dag:

        # To be implemented by the customer.
        # Retrieve the timestamp from the prior run
        start_hwm = 'YYYY-MM-DD HH:mm:SS'
        end_hwm = 'YYYY-MM-DD HH:mm:SS' # timestamp now

        ext = Extractor(**KADA_EXTRACTOR_CONFIG)

        start = DummyOperator(task_id="start")

        with TaskGroup("taskgroup_1", tooltip="extract tableau and upload") as extract_upload:
            task_1 = PythonOperator(
                task_id="extract_tableau",
                python_callable=ext.run,
                op_kwargs={"start_hwm": start_hwm, "end_hwm": end_hwm},
                provide_context=True,
            )

            task_2 = PythonOperator(
                task_id="upload_extracts",
                python_callable=upload,
                op_kwargs={},
                provide_context=True,
            )

            # To be implemented by the customer.
            # Timestamp needs to be saved for next run
            task_3 = DummyOperator(task_id='save_hwm')

        end = DummyOperator(task_id='end')

        start >> extract_upload >> end

Last updated: March 21, 2026

---
language: "en"
---
# Bytehouse (via Direct Connect method)

This page will walkthrough the setup of Bytehouse in K using the direct connect method.

## Integration details

|       **Scope**        | **Included** | **Comments** |
|------------------------|--------------|--------------|
| Metadata               | YES          |              |
| Lineage                | YES          |              |
| Usage                  | No           |              |
| Sensitive Data Scanner | N/A          |              |

*** ** * ** ***

## Step 1: Establish Bytehouse Access

The service user/account/role will require permissions to the following:

* Refer to Bytehouse documentation for required permissions

After this step you should have the following information:

* Tenant Account ID

* Role

* Key

* Secret

*** ** * ** ***

## Step 2: Create the Source in K

Create a Bytehouse source in K.

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Click **Add Source** and select **Bytehouse**

* Select **Direct Connect** and add your Bytehouse **Source Settings**

  * **Name:** Give the Bytehouse source a name in K.

  * **Host:** Enter a hostname for your Bytehouse instance

  * **Tenant Account ID:** You can locate the Account ID in the Tenant Management tab

  * **Gateway** and **Gateway Port:** Refer to [Supported Cloud Providers and Regions - ByteHouse Byteplus](https://docs.byteplus.com/en/docs/bytehouse/docs-supported-regions-and-providers) for your Gateway details

  * Confirm if you want to:

    * Enable data masking

    * Extract meta only

* Click **Save \& Next**

* Setup your API connection

  * Add your API Key or Token

* **Test your connection** and click **Next**

*** ** * ** ***

## Step 3: Schedule Bytehouse source load

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Locate your new Bytehouse Source and click on the **Schedule Settings** (clock) icon to set the schedule

*** ** * ** ***

## Step 4: Manually run an ad hoc load to test Bytehouse

Last updated: July 17, 2026

---
language: "en"
---
# Cascade Update

Updated in Version 6.0

Sometimes you may want to update a property for all columns and tables associated with a schema. Cascade Update is a quick way to perform this type of bulk update from a parent asset downwards.

**Example:** When you perform a Cascade Update on a schema, all tables and columns that are part of that schema will be updated.

**Access:** Only Data Governance Managers, Data Managers, and Admin users have access to Cascade Update.

*** ** * ** ***

## How to perform a Cascade Update

Navigate to the **Profile Page** of the parent asset you want to update (e.g. a database, schema, table, or report).

* **Step 1)** Click the **Options** button at the top right corner.

* **Step 2)** Click **Cascade Update**.

![image-20260531-110527.png](https://docs.kada.ai/__attachments/a_6853fe41a2a75d934740d53d4e475fcb9140563c288d6f5e6f56bb34c055a316/image-20260531-110527.png?cb=4e0c9fa3245315e55fdec656ebac375b)

* **Step 3)** Select all of the properties you would like to update and confirm the changes you want to make.

  ![image-20260531-110656.png](https://docs.kada.ai/__attachments/a_8863da7b0ab78ef82ba7c3f11a8949c50990b65ae7e42e4bb74c355dfbcf63d1/image-20260531-110656.png?cb=3964d4ae5d26cb0ccfe852e84e631871)
* **Step 4)** Click **Next**.

* **Step 5)** Review the changes and the data assets that will be updated. When you are ready, click **Submit**.

* **Step 6)** Your Cascade Update job will be submitted.

  ![image-20260531-110808.png](https://docs.kada.ai/__attachments/a_b907dc678763e827710ac7eeaab0e792ab7f3b7b300c3c2db58bc7a7dd154986/image-20260531-110808.png?cb=c47a97e4ef50b958b9f125b1918d66ca)

*** ** * ** ***

## Viewing previous Cascade Update jobs

You can view all previous Cascade Update jobs by navigating to the **Actions Page**  
![image-20260531-111104.png](https://docs.kada.ai/__attachments/a_66bc1e995fb46678fd34815d96a7202cce4300cee0d6340b378545996cc04e48/image-20260531-111104.png?cb=e0f87c0c2de54f50518a51ef669e3cff)

Last updated: July 26, 2026

---
language: "en"
---
# Change History

Updated in Version 6.0

K helps you catalog and identify changes to data items through automated change detection. On each data profile page you can click on the **Changes** tab to view a timeline of changes detected.

*** ** * ** ***

## Automated Change Detection

When data and content items change in structure (such as a column being added to a Table or a sheet being added to a Report), K will catalog the change and notify impacted users automatically.

To access the Change History, click on the **menu** icon and then **Change history**.  
![image-20260312-122145.png](https://docs.kada.ai/__attachments/a_75965e524824667d720e165f0ae2434f8f781caa1fa7584e1b5e6e689d480f8c/image-20260312-122145.png?cb=019713d23dcf5de81159ed6be7867353)

For each change that is detected, you can click in the **Note** section and add details about the change to help other users understand the context of the change.  
![image-20260312-122246.png](https://docs.kada.ai/__attachments/a_140a4088f6964df75c54d0d48a2ecff1123cafce5c1f8a0985ef580d65e1fb36/image-20260312-122246.png?cb=f682ede7f712715dea06ae9cac746f1d)

*** ** * ** ***

## Types of Change Automatically Detected

K has the ability to detect the following types of change:  

| **Type of Data Asset** |                                                                                                                                                  **Changes Detected**                                                                                                                                                   |
|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Table                  | * Table added * Column added * Column modified * Column deleted * Table deleted                                                                                                                                                                                                                                         |
| Column                 | * Column first created * Column deleted * Column type modified                                                                                                                                                                                                                                                          |
| Report                 | * Report first created * Report deleted * Page in the report is Added * Page in the report is Deleted * Report name change                                                                                                                                                                                              |
| Sheet                  | * Sheet is first created * Sheet is deleted                                                                                                                                                                                                                                                                             |
| Dataset                | * New Dataset Table added * Dataset table deleted * Dataset table modified                                                                                                                                                                                                                                              |
| Dataset Table          | * Dataset Field Added * Dataset Field Deleted * Dataset Field Modified                                                                                                                                                                                                                                                  |
| Dataset Field          | * Field first created * Field deleted * Type is modified * Calculation is modified                                                                                                                                                                                                                                      |
| Schema                 | * Table added * Table removed                                                                                                                                                                                                                                                                                           |
| K Metadata             | * K User updates to description and properties to a data profile                                                                                                                                                                                                                                                        |
| K Linkage              | * All changes to Knowledge (e.g. new Decisions, Business Logic, How To guides) * Additions and removals of any linkages to collections. Examples include * Owners and stewards added/removed * Collection instances added/removed * Tags added/removed * Classifications, Verified use cases, Domains etc added.removed |

*** ** * ** ***

## Metadata Change Management

When you update a description, add collection links, change status, or make other profile changes, K catalogues those changes and notifies relevant users.

These appear in the Timeline tab under **K metadata changes** and **K linkage changes**.

Last updated: July 26, 2026

---
language: "en"
---
# Clickhouse (via Collector method)

This page outlines the ClickHouse Collector versions that are available.

We always recommend using the latest Collector Method to ensure that you can access the latest features.

*** ** * ** ***

## Integration details

|       **Scope**        | **Included** | **Comments** |
|------------------------|--------------|--------------|
| Metadata               | YES          |              |
| Lineage                | YES          |              |
| Usage                  | NO           |              |
| Sensitive Data Scanner | N/A          |              |

*** ** * ** ***

## ClickHouse Version History

|                          **Version Number**                           | **Release Month** | **K Platform Compatibility** | **Collector Core Library Compatibility** |  **Release changes**   |
|-----------------------------------------------------------------------|-------------------|------------------------------|------------------------------------------|------------------------|
| [V3.0.0](https://docs.kada.ai/k-knowledge-base/clickhouse-via-collector-method-v3-0-0.md) | Sept 2024         | 5.40 - 5.42                  | 1.1.5 - 1.1.9                            | First version released |

Last updated: March 12, 2026

---
language: "en"
---
# Clickhouse (via Collector method) - v3.0.0

## About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

### Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

2. You want to push metadata to KADA rather than allow it to pull data for security reasons

3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

1. Deploying and orchestrating the extract code

2. Managing a high water mark so the extract only pulls the latest metadata

3. Storing and pushing the extracts to your K instance

*** ** * ** ***

## Pre-requisites

**Collector Server Minimum Requirements**

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

* CPU: 2 vCPU

* Memory: 8GB

* Storage: 30GB (depends on historical data extracted)

* OS: unix distro e.g. RHEL preferred but can also work with Windows Server

* Python 3.10.x or later

* Access to [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md)

**ClickHouse Requirements**

* Access to the following tables

  1. `system.databases`

  2. `system.tables`

  3. `system.columns`

*** ** * ** ***

## Step 1: Enabling logging

Enable logging in Clickhouse.

*** ** * ** ***

## Step 2: Create the Source in K

Create a ClickHouse source in K

* Go to **Settings** , Select **Sources** and click **Add Source**

* Select **"Load from File system" option**

* Give the source a **Name** - e.g. ClickHouse Production

* Add the **Host name** for the ClickHouse Server

* Click **Finish Setup**

*** ** * ** ***

## Step 3: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

1. Go to Platform Settings - Settings. Note down the value of this setting:

   * If using Azure: **storage_azure_storage_account**

   * If using AWS:

     * **storage_root_folder** - the AWS s3 bucket

     * **storage_aws_region** - the region where the AWS s3 bucket is hosted

2. Go to Sources - Edit the Source you have configured. Note down the **landing directory** in the About this Source section.

To connect to the landing directory you will need:

* If using Azure: a **SAS token** to push data to the landing directory. Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

* If using AWS:

  * An **Access key and Secret** . Request this from KADA Support ([support@kada.ai](mailto:support@kada.ai))

  * OR provide your IAM role to KADA Support to provision access.

*** ** * ** ***

## Step 4: Install the Collector

You can download the latest Core Library and whl via **Platform Settings → Sources** → **Download Collectors**

Run the following command to install the collector

    pip install kada_collectors_extractors_<version>-none-any.whl

You will also need to install the common library kada_collectors_lib for this collector to function properly.

    pip install kada_collectors_lib-<version>-none-any.whl

*** ** * ** ***

## Step 5: Configure the Collector

The ClickHouse collector only extracts metadata and does not extract or process query usage on the database.  

|   **FIELD**   | **FIELD TYPE** |                            **DESCRIPTION**                             |                   **EXAMPLE**                    |
|---------------|----------------|------------------------------------------------------------------------|--------------------------------------------------|
| username      | string         | Username to log into ClickHouse                                        | "myuser"                                         |
| password      | string         | Password to log into ClickHouse                                        | "password"                                       |
| server        | string         | ClickHouse instance server                                             | "t1x6j03yyo.ap-southeast-2.aws.clickhouse.cloud" |
| port          | integer        | The port to connect to the ClickHouse instance, generally this is 9440 | 9440                                             |
| host          | string         | The onboarded host in K for the ClickHouse Source                      | "t1x6j03yyo.ap-southeast-2.aws.clickhouse.cloud" |
| database_name | string         | The onboarded database name in K for the ClickHouse Source             | "myclickhouse"                                   |
| meta_only     | boolean        | Currently we only support meta only as true                            | true                                             |
| output_path   | string         | Absolute path to the output location                                   | "/tmp/output"                                    |
| mask          | boolean        | To enable masking or not                                               | true                                             |
| compress      | boolean        | To enable compression or not to .csv.gz                                | true                                             |
| timeout       | integer        | Timeout setting in seconds                                             | 80000                                            |

**kada_clickhouse_extractor_config.json**

    {
        "username": "",
        "password": "",
        "server": "",
        "port": 9440,
        "database_name": "",
        "host": "",
        "output_path": "/tmp/output",
        "mask": true,
        "compress": true,
        "meta_only": true,
        "timeout": 80000
    }

*** ** * ** ***

## Step 6: Run the Collector

This is the wrapper script: **kada_clickhouse_extractor.py**

    import os
    import argparse
    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.clickhouse import Extractor

    get_generic_logger('root')

    _type = 'clickhouse'
    dirname = os.path.dirname(__file__)
    filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

    parser = argparse.ArgumentParser(description='KADA Clickhouse Extractor.')
    parser.add_argument('--config', '-c', dest='config', default=filename)
    parser.add_argument('--name', '-n', dest='name', default=_type)
    args = parser.parse_args()

    start_hwm, end_hwm = get_hwm(args.name)

    ext = Extractor(**load_config(args.config))
    ext.test_connection()
    ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

    publish_hwm(args.name, end_hwm)

*** ** * ** ***

## Step 7: Check the Collector Outputs

**K Extracts**

A set of files (eg metadata, databaselog, linkages, events etc) will be generated in the output_path directory.

**High Water Mark File**

A high water mark file is created called **clickhouse_hwm.txt**.

Refer to [Collector Integration General Notes](https://docs.kada.ai/k-knowledge-base/collector-integration-general-notes.md) for more information.

*** ** * ** ***

## Step 8: Push the Extracts to K

Once the files have been validated, you can push the files to the [K landing directory](https://docs.kada.ai/k-knowledge-base/how-to-upload-a-file-to-the-k-landing-directory.md).

*** ** * ** ***

## Example: Using Airflow to orchestrate the Extract and Push to K

The following example is how you can orchestrate the Tableau collector using Airflow and push the files to K hosted on Azure. The code is not expected to be used as-is but as a template for your own DAG.
Python

    # built-in
    import os

    # Installed
    from airflow.operators.python_operator import PythonOperator
    from airflow.models.dag import DAG
    from airflow.operators.dummy import DummyOperator
    from airflow.utils.dates import days_ago
    from airflow.utils.task_group import TaskGroup

    from plugins.utils.azure_blob_storage import AzureBlobStorage

    from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
    from kada_collectors.extractors.tableau import Extractor

    # To be configured by the customer.
    # Note variables may change if using a different object store.
    KADA_SAS_TOKEN = os.getenv("KADA_SAS_TOKEN")
    KADA_CONTAINER = ""
    KADA_STORAGE_ACCOUNT = ""
    KADA_LANDING_PATH = "lz/tableau/landing"
    KADA_EXTRACTOR_CONFIG = {
        "server_address": "http://tabserver",
        "username": "user",
        "password": "password",
        "sites": [],
        "db_host": "tabserver",
        "db_username": "repo_user",
        "db_password": "repo_password",
        "db_port": 8060,
        "db_name": "workgroup",
        "meta_only": False,
        "retries": 5,
        "dry_run": False,
        "output_path": "/set/to/output/path",
        "mask": True,
        "mapping": {}
    }

    # To be implemented by the customer.
    # Upload to your landing zone storage.
    # Change '.csv' to '.csv.gz' if you set compress = true in the config
    def upload():
      output = KADA_EXTRACTOR_CONFIG['output_path']
      for filename in os.listdir(output):
          if filename.endswith('.csv'):
            file_to_upload_path = os.path.join(output, filename)

            AzureBlobStorage.upload_file_sas_token(
                client=KADA_SAS_TOKEN,
                storage_account=KADA_STORAGE_ACCOUNT,
                container=KADA_CONTAINER,
                blob=f'{KADA_LANDING_PATH}/{filename}',
                local_path=file_to_upload_path
            )

    with DAG(dag_id="taskgroup_example", start_date=days_ago(1)) as dag:

        # To be implemented by the customer.
        # Retrieve the timestamp from the prior run
        start_hwm = 'YYYY-MM-DD HH:mm:SS'
        end_hwm = 'YYYY-MM-DD HH:mm:SS' # timestamp now

        ext = Extractor(**KADA_EXTRACTOR_CONFIG)

        start = DummyOperator(task_id="start")

        with TaskGroup("taskgroup_1", tooltip="extract tableau and upload") as extract_upload:
            task_1 = PythonOperator(
                task_id="extract_tableau",
                python_callable=ext.run,
                op_kwargs={"start_hwm": start_hwm, "end_hwm": end_hwm},
                provide_context=True,
            )

            task_2 = PythonOperator(
                task_id="upload_extracts",
                python_callable=upload,
                op_kwargs={},
                provide_context=True,
            )

            # To be implemented by the customer.
            # Timestamp needs to be saved for next run
            task_3 = DummyOperator(task_id='save_hwm')

        end = DummyOperator(task_id='end')

        start >> extract_upload >> end

Last updated: March 21, 2026

---
language: "en"
---
# Clickhouse (via Direct Connect method)

This page will walkthrough the setup of Clickhouse in K using the direct connect method.

## Integration details

|       **Scope**        | **Included** | **Comments** |
|------------------------|--------------|--------------|
| Metadata               | YES          |              |
| Lineage                | YES          |              |
| Usage                  | No           |              |
| Sensitive Data Scanner | N/A          |              |

*** ** * ** ***

## Step 1: Establish Clickhouse Access

The service user/account/role will require permissions to the following:

* Refer to Clickhouse documentation for required permissions

After this step you should have the following information:

* Tenant Account ID

* Role

* Key

* Secret

*** ** * ** ***

## Step 2: Create the Source in K

Create a Clickhouse source in K.

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Click **Add Source** and select **Clickhouse**

* Select **Direct Connect** and add your Clickhouse **Source Settings**

  * **Name:** Give the Clickhouse source a name in K.

  * **Host:** Enter a hostname for your Clickhouse instance

  * **Port:** Refer to [Network ports \| ClickHouse Docs](https://clickhouse.com/docs/en/guides/sre/network-ports) for your Port details

  * Confirm if you want to:

    * Enable data masking

    * Extract meta only

* Click **Save \& Next**

* Setup your Connection details:

  * **Host**

  * **Username** and **Password**

* **Test your connection** and click **Next**

*** ** * ** ***

## Step 3: Schedule Clickhouse source load

* Select **Platform Settings** in the side bar

* In the pop-out side panel, under **Integrations** click on **Sources**

* Locate your new Clickhouse Source and click on the **Schedule Settings** (clock) icon to set the schedule

*** ** * ** ***

## Step 4: Manually run an ad hoc load to test Clickhouse

Last updated: July 17, 2026

[Next Page](https://docs.kada.ai/llms-full.txt/1)