Qlik Sense (via Collector method)

About Collectors

Collectors are extractors that are developed and managed by you (a customer of K).

KADA provides python libraries that customers can use to quickly deploy a Collector.

Why you should use a Collector

There are several reasons why you may use a collector vs the direct connect extractor:

  1. You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions

  2. You want to push metadata to KADA rather than allow it to pull data for security reasons

  3. You want to inspect the metadata before pushing it to K

Using a collector requires you to manage:

  1. Deploying and orchestrating the extract code

  2. Managing a high water mark so the extract only pulls the latest metadata

  3. Storing and pushing the extracts to your K instance


Integration details

This collector supports Qlik Sense Enterprise client-managed (on-premise). Qlik Cloud and QlikView are not supported by this collector.

Scope

Included

Comments

Metadata

YES

Streams, apps, sheets, data models (tables and fields), load scripts, QVD files, reload tasks

Lineage

YES

Database tables, files and QVDs → app data model → sheets, from the archived reload logs. QVD → app → QVD chains across apps

Usage

YES

Sheet and app views and downloads, from the Engine audit logs. App reloads, from the archived reload logs

Users

YES

From the Qlik repository

Sensitive Data Scanner

N/A



Pre-requisites

Collector Server Minimum Requirements

For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:

  • CPU: 2 vCPU

  • Memory: 8GB

  • Storage: 30GB (depends on historical data extracted)

  • OS: unix distro e.g. RHEL preferred but can also work with Windows Server

  • Python 3.10.x or later

  • Access to K landing directory

  • A Windows server joined to the same domain as Qlik Sense. The collector reads the Qlik log archive and the QVD folders over Windows file shares (UNC paths such as \\server\share\...)

  • Python 3.10 or later

  • Network access from the collector server to the Qlik Sense central node

    • Port 4242 (Qlik Repository Service, certificate authentication)

    • Port 4747 (Qlik Engine, certificate authentication)

    • These are Qlik's certificate ports, not the hub (443) or the proxy (4243), and are often closed by default

  • Enough free disk space in the collector's working folder for app exports. Each app is exported without data while the collector runs and the files are deleted at the end of the run



Qlik Sense Requirements

Qlik user access

  • Client and root certificates exported from the QMC. This needs a user with the RootAdmin role, once, during setup

    • In the QMC go to Certificates, add the machine name of the collector server, optionally set a password, tick Include secret key, choose Windows format and click Export

    • You will need for the setup

      • client.pfx (and its password, if one was set)

      • root.cer

    • The certificate identifies the collector server to Qlik. Keep it secure: it grants access to the Qlik Repository and Engine

  • A Qlik service account for the collector to run as. Every call the collector makes runs as this user

    • You will need for the setup

      • User directory, e.g. DOMAIN

      • User ID, e.g. svc_kada

    • The account needs the ContentAdmin role. RootAdmin and HubAdmin are not required.

      ContentAdmin gives the account the following access:

Access

Used for

Read apps and sheets in the streams to be collected

Apps, sheets and their metadata

Export apps

The app load script, from an export without data

Read data connections

Where QVDs are stored and which databases apps connect to

Read users

Users and their names

Read reload tasks

Reload task names and status

Open apps in the Engine

The app data model (tables and fields)

  • The hostname of the Qlik Sense central node, exactly as it appears on the Qlik server certificate. This is often the short server name (e.g. qlikprd01) rather than the hub URL, and a full domain name may fail the certificate check


Qlik logging

  • Engine audit logging must be enabled (QMC → Engines → Logging → Audit activity log level not set to Off) for usage to be captured


Collector user Log and File access

The windows account (not the Qlik service account) will need the following:

  • Read access to the Qlik ArchivedLogs folder, e.g. \\<server>\QlikShare\ArchivedLogs

    • Reload logs: <Node>\Script\

    • Engine audit logs: <Node>\Engine\Audit\

  • Read access to every folder that holds QVD files, i.e. the paths of the folder data connections in the QMC


Optional Requirements

  • To link apps to database sources that are reached through an ODBC DSN (e.g. ODBC CONNECT TO [prod_dwh]): add the DSN to the collector mapping, see Step 4.

    A DSN does not name the database host, so without a mapping that lineage cannot be joined to the database source in K


Step 1: Create the Source in K

Create a Qlik Sense source in K

  • Go to Settings, Select Sources and click Add Source

  • Select "Load from File" option

  • Give the source a Name - e.g. Qlik Sense Production

  • Add the Host name for the Qlik Sense hub, e.g. qlik.company.com

  • Click Next & Finish Setup


Step 2: Getting Access to the Source Landing Directory

When using a Collector you will push metadata to a K landing directory.

To find your landing directory you will need to:

  1. Go to Platform Settings - Settings. Note down the value of this setting:

    • If using Azure: storage_azure_storage_account

    • If using AWS:

      • storage_root_folder - the AWS s3 bucket

      • storage_aws_region - the region where the AWS s3 bucket is hosted

  2. Go to Sources - Edit the Source you have configured. Note down the landing directory in the About this Source section.

To connect to the landing directory you will need:

  • If using Azure: a SAS token to push data to the landing directory. Request this from KADA Support (support@kada.ai)

  • If using AWS:

    • An Access key and Secret. Request this from KADA Support (support@kada.ai)

    • OR provide your IAM role to KADA Support to provision access.


Step 3: Install the Collector

It is recommended to use a python environment such as pyenv or pipenv if you are not intending to install this package at the system level.

You can download the latest Core Library and Qlik whl via Platform Settings → Sources → Download Collectors

Run the following command to install the collector

pip install kada_collectors_extractors_qlik-1.0.0-py3-none-any.whl

You will also need to install the corresponding common library kada_collectors_lib-x.x.x for this collector to function properly.

pip install kada_collectors_lib-x.x.x-py3-none-any.whl


Step 4: Configure the Collector

FIELD

FIELD TYPE

DESCRIPTION

EXAMPLE

server

string

Hostname of the Qlik Sense central node, exactly as on the Qlik server certificate. See Pre-requisites

"qlikprd01"

client_pfx

string

Path to the client certificate exported from the QMC

"C:/kada/certs/client.pfx"

client_pfx_password

string

Password of the client certificate, empty if none was set

""

root_cer

string

Path to the root certificate exported from the QMC

"C:/kada/certs/root.cer"

user_directory

string

User directory of the Qlik service account

"OKTA"

user_id

string

User ID of the Qlik service account

"svc_kada"

api_host

string

Optional. The public hub hostname, used to build links back to Qlik. Defaults to server

"qlik.company.com"

log_root

string

Path to the ArchivedLogs folder

"//qlikprd01/QlikShare/ArchivedLogs"

audit_root

string

Optional. Path to the Engine audit logs, if they are archived somewhere other than log_root. Empty uses log_root

""

audit_max_age_days

integer

Optional. Only read Engine audit logs modified in the last this many days. Applies to the first run, or a run after a long gap: every other run only reads audit logs since the previous run. 0 reads the whole archive. Default is 90

90

stream_id

string

Optional. Comma separated ids of the streams to collect. Empty collects every published app. Apps outside these streams that feed them are still collected

""

mapping

json

Optional. Maps ODBC DSNs and connection names to a host and database onboarded in K

{ "prod_dwh": { "host": "sqlprd01", "database": "dwh" } }

output_path

string

Absolute path to the output location where files are to be written. Its contents are deleted at the start of every run

"C:/kada/output"

mask

boolean

To enable masking or not. Note the masker is designed for SQL: on a Qlik load script it removes literal values but flattens the script to a single line

false

compress

boolean

To gzip the output or not

true

These parameters can be added directly into the run or you can pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.

kada_qlik_extractor_config.json

JSON
{
    "server": "",
    "client_pfx": "",
    "client_pfx_password": "",
    "root_cer": "",
    "user_directory": "",
    "user_id": "",
    "api_host": "",
    "log_root": "",
    "audit_root": "",
    "audit_max_age_days": 90,
    "stream_id": "",
    "mapping": {},
    "output_path": "C:/kada/output",
    "mask": false,
    "compress": true
}

Windows paths in the JSON can be written with forward slashes, e.g. "//qlikprd01/QlikShare/ArchivedLogs", or with every backslash doubled, e.g. "\\\\qlikprd01\\QlikShare\\ArchivedLogs".


Step 5: Run the Collector

The following code is an example of how to run the extractor. You may need to uplift this code to meet any code standards at your organisation.

This can be executed in any python environment where the whl has been installed.

This is the wrapper script: kada_qlik_extractor.py

Python
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.qlik import Extractor

get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger

_type = 'qlik'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))

parser = argparse.ArgumentParser(description='KADA Qlik Sense Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()

start_hwm, end_hwm = get_hwm(args.name)

ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})

# ext.next_hwm, not end_hwm
# Qlik's reload history comes from an archived log SHARE, which is routinely behind
# the estate (the extractor reports this as "the log share is BEHIND"). Advancing
# the watermark to end_hwm would step over reloads that reach the share after this
# run, and they would never be read again. next_hwm is the newest event actually
# emitted, so a lagging archive is simply re-read next time.
publish_hwm(args.name, ext.next_hwm)

Important: publish ext.next_hwm, not end_hwm as the other collector wrappers do. The archived logs can arrive on the share later than the reloads and views they record, and publishing end_hwm would skip them permanently.

test_connection checks both the Qlik Repository (port 4242) and the Engine (port 4747), as they are often opened separately on the firewall.

The collector opens every app in scope in the Qlik Engine to read its data model. On a large estate, we recommend scheduling the collector outside business hours.

Advance options:

If you wish to maintain your own high water mark files elsewhere you can use the above section's script as a guide on how to call the extractor. The configuration file is simply the keyword arguments in JSON format. Refer to Collector Integration General Notes | Storing High Water Marks (HWM) for more information.

Python
from kada_collectors.extractors.qlik import Extractor

kwargs = {}  # However you choose to construct your args
hwm_kwargs = {"start_hwm": "YYYY-MM-DD HH:MM:SS", "end_hwm": "YYYY-MM-DD HH:MM:SS"}  # UTC

ext = Extractor(**kwargs)
ext.run(**hwm_kwargs)
next_hwm = ext.next_hwm  # Store this as the start_hwm of the next run

class Extractor(server: str = None, client_pfx: str = None, root_cer: str = None, \
    user_directory: str = None, user_id: str = None, client_pfx_password: str = '', \
    api_host: str = None, log_root: str = None, audit_root: str = '', stream_id: str = '', \
    output_path: str = './output', compress: bool = False, mask: bool = False, \
    mapping: dict = None, audit_max_age_days: int = 90) -> None


Step 6: Check the Collector Outputs

K Extracts

A set of files will be generated in the output_path directory:

File

Contents

META

Streams, apps, sheets, data models, tables, fields, QVD files, load scripts and SQL queries, reloads

LINKAGES

Lineage between databases, files, QVDs, apps and sheets

PROPERTIES

Additional app and reload properties, e.g. reload task status and last reload time

USERS

Qlik users

EVENTS

Views and downloads (audit logs), reloads (reload logs), QVD writes, and app / sheet creation and modification

Metadata and lineage are extracted in full on every run. Only events are limited to the period since the last run.

High Water Mark File

A high water mark file is created in the same directory as the execution called qlik_hwm.txt. This file is only produced if you call the publish_hwm method.


Step 7: Push the Extracts to K

Once the files have been validated, you can push the files to the K landing directory.


Known limitations

  • Lineage comes from the archived reload logs. An app's lineage reflects its most complete reload in the log archive. An app with no reload log in the archive has no lineage.

  • Apps using section access may refuse the service account in the Engine. Their tables and fields are then not collected, and their lineage comes from the reload log only.

  • Unpublished apps (in users' personal work areas) are not collected.

  • Apps not reloaded in 90 days keep their metadata and lineage, but are no longer shown as the producer of the QVDs they once wrote.

  • Date stamped QVD files (e.g. sales_20260131.qvd, sales_20260201.qvd) are shown as a single QVD.

  • Views and downloads history on the first run goes back 90 days by default. Set audit_max_age_days to read further back, as far as the audit logs are kept in the archive.

  • ODBC DSN connections need a mapping entry to join to the database source in K.

Troubleshooting

Message

Cause / fix

The following must be populated: ...

A required config field is empty

client_pfx does not exist or is not a file / root_cer ...

The certificate path is wrong, or not readable by the account running the collector

log_root does not exist or is not a directory

The ArchivedLogs path is wrong, or the Windows account running the collector has no access to the share

audit_max_age_days must be a whole number of days

audit_max_age_days is negative, a decimal, or in quotes in the config. Use a whole number such as 90, or 0 for no limit

Unable to connect to the QRS API at ...

Port 4242 is closed, server does not match the name on the Qlik server certificate, or the certificates are not the ones exported for this collector server

QRS ... returned no apps

The service account cannot see any apps (check it has the ContentAdmin role), or stream_id names streams with no apps

Unable to connect to the Engine at wss://...:4747

Port 4747 is closed, or the service account cannot open apps in the Engine

engine skip <app>: ...

That app could not be opened in the Engine, usually because of section access. Its tables and fields are skipped, lineage still comes from its reload log

QVDs reported as Access denied or Share unreachable

The Windows account running the collector has no read access to that share

unknown_host warnings

An app connects through an ODBC DSN or connection with no host. Add it to mapping

Last updated: