About Collectors
Collectors are extractors that are developed and managed by you (a customer of K).
KADA provides python libraries that customers can use to quickly deploy a Collector.
Why you should use a Collector
There are several reasons why you may use a collector vs the direct connect extractor:
-
You are using the KADA SaaS offering and it cannot connect to your sources due to firewall restrictions
-
You want to push metadata to KADA rather than allow it to pull data for security reasons
-
You want to inspect the metadata before pushing it to K
Using a collector requires you to manage:
-
Deploying and orchestrating the extract code
-
Managing a high water mark so the extract only pulls the latest metadata
-
Storing and pushing the extracts to your K instance
Integration details
This collector supports Qlik Sense Enterprise client-managed (on-premise). Qlik Cloud and QlikView are not supported by this collector.
|
Scope |
Included |
Comments |
|---|---|---|
|
Metadata |
YES |
Streams, apps, sheets, data models (tables and fields), load scripts, QVD files, reload tasks |
|
Lineage |
YES |
Database tables, files and QVDs → app data model → sheets, from the archived reload logs. QVD → app → QVD chains across apps |
|
Usage |
YES |
Sheet and app views and downloads, from the Engine audit logs. App reloads, from the archived reload logs |
|
Users |
YES |
From the Qlik repository |
|
Sensitive Data Scanner |
N/A |
|
Pre-requisites
Collector Server Minimum Requirements
For the collector to operate effectively, it will need to be deployed on a server with the below minimum specifications:
-
CPU: 2 vCPU
-
Memory: 8GB
-
Storage: 30GB (depends on historical data extracted)
-
OS: unix distro e.g. RHEL preferred but can also work with Windows Server
-
Python 3.10.x or later
-
Access to K landing directory
-
A Windows server joined to the same domain as Qlik Sense. The collector reads the Qlik log archive and the QVD folders over Windows file shares (UNC paths such as
\\server\share\...) -
Python 3.10 or later
-
Network access from the collector server to the Qlik Sense central node
-
Port 4242 (Qlik Repository Service, certificate authentication)
-
Port 4747 (Qlik Engine, certificate authentication)
-
These are Qlik's certificate ports, not the hub (443) or the proxy (4243), and are often closed by default
-
-
Enough free disk space in the collector's working folder for app exports. Each app is exported without data while the collector runs and the files are deleted at the end of the run
Qlik Sense Requirements
Qlik user access
-
Client and root certificates exported from the QMC. This needs a user with the
RootAdminrole, once, during setup-
In the QMC go to Certificates, add the machine name of the collector server, optionally set a password, tick Include secret key, choose Windows format and click Export
-
You will need for the setup
-
client.pfx(and its password, if one was set) -
root.cer
-
-
The certificate identifies the collector server to Qlik. Keep it secure: it grants access to the Qlik Repository and Engine
-
-
A Qlik service account for the collector to run as. Every call the collector makes runs as this user
-
You will need for the setup
-
User directory, e.g.
DOMAIN -
User ID, e.g.
svc_kada
-
-
The account needs the ContentAdmin role.
RootAdminandHubAdminare not required.
ContentAdmin gives the account the following access:
-
|
Access |
Used for |
|---|---|
|
Read apps and sheets in the streams to be collected |
Apps, sheets and their metadata |
|
Export apps |
The app load script, from an export without data |
|
Read data connections |
Where QVDs are stored and which databases apps connect to |
|
Read users |
Users and their names |
|
Read reload tasks |
Reload task names and status |
|
Open apps in the Engine |
The app data model (tables and fields) |
-
The hostname of the Qlik Sense central node, exactly as it appears on the Qlik server certificate. This is often the short server name (e.g.
qlikprd01) rather than the hub URL, and a full domain name may fail the certificate check
Qlik logging
-
Engine audit logging must be enabled (QMC → Engines → Logging → Audit activity log level not set to Off) for usage to be captured
Collector user Log and File access
The windows account (not the Qlik service account) will need the following:
-
Read access to the Qlik ArchivedLogs folder, e.g.
\\<server>\QlikShare\ArchivedLogs-
Reload logs:
<Node>\Script\ -
Engine audit logs:
<Node>\Engine\Audit\
-
-
Read access to every folder that holds QVD files, i.e. the paths of the folder data connections in the QMC
Optional Requirements
-
To link apps to database sources that are reached through an ODBC DSN (e.g.
ODBC CONNECT TO [prod_dwh]): add the DSN to the collectormapping, see Step 4.
A DSN does not name the database host, so without a mapping that lineage cannot be joined to the database source in K
Step 1: Create the Source in K
Create a Qlik Sense source in K
-
Go to Settings, Select Sources and click Add Source
-
Select "Load from File" option
-
Give the source a Name - e.g. Qlik Sense Production
-
Add the Host name for the Qlik Sense hub, e.g.
qlik.company.com -
Click Next & Finish Setup
Step 2: Getting Access to the Source Landing Directory
When using a Collector you will push metadata to a K landing directory.
To find your landing directory you will need to:
-
Go to Platform Settings - Settings. Note down the value of this setting:
-
If using Azure: storage_azure_storage_account
-
If using AWS:
-
storage_root_folder - the AWS s3 bucket
-
storage_aws_region - the region where the AWS s3 bucket is hosted
-
-
-
Go to Sources - Edit the Source you have configured. Note down the landing directory in the About this Source section.
To connect to the landing directory you will need:
-
If using Azure: a SAS token to push data to the landing directory. Request this from KADA Support (support@kada.ai)
-
If using AWS:
-
An Access key and Secret. Request this from KADA Support (support@kada.ai)
-
OR provide your IAM role to KADA Support to provision access.
-
Step 3: Install the Collector
It is recommended to use a python environment such as pyenv or pipenv if you are not intending to install this package at the system level.
You can download the latest Core Library and Qlik whl via Platform Settings → Sources → Download Collectors
Run the following command to install the collector
pip install kada_collectors_extractors_qlik-1.0.0-py3-none-any.whl
You will also need to install the corresponding common library kada_collectors_lib-x.x.x for this collector to function properly.
pip install kada_collectors_lib-x.x.x-py3-none-any.whl
Step 4: Configure the Collector
|
FIELD |
FIELD TYPE |
DESCRIPTION |
EXAMPLE |
|---|---|---|---|
|
server |
string |
Hostname of the Qlik Sense central node, exactly as on the Qlik server certificate. See Pre-requisites |
"qlikprd01" |
|
client_pfx |
string |
Path to the client certificate exported from the QMC |
"C:/kada/certs/client.pfx" |
|
client_pfx_password |
string |
Password of the client certificate, empty if none was set |
"" |
|
root_cer |
string |
Path to the root certificate exported from the QMC |
"C:/kada/certs/root.cer" |
|
user_directory |
string |
User directory of the Qlik service account |
"OKTA" |
|
user_id |
string |
User ID of the Qlik service account |
"svc_kada" |
|
api_host |
string |
Optional. The public hub hostname, used to build links back to Qlik. Defaults to server |
"qlik.company.com" |
|
log_root |
string |
Path to the ArchivedLogs folder |
|
|
audit_root |
string |
Optional. Path to the Engine audit logs, if they are archived somewhere other than log_root. Empty uses log_root |
"" |
|
audit_max_age_days |
integer |
Optional. Only read Engine audit logs modified in the last this many days. Applies to the first run, or a run after a long gap: every other run only reads audit logs since the previous run. 0 reads the whole archive. Default is 90 |
90 |
|
stream_id |
string |
Optional. Comma separated ids of the streams to collect. Empty collects every published app. Apps outside these streams that feed them are still collected |
"" |
|
mapping |
json |
Optional. Maps ODBC DSNs and connection names to a host and database onboarded in K |
|
|
output_path |
string |
Absolute path to the output location where files are to be written. Its contents are deleted at the start of every run |
"C:/kada/output" |
|
mask |
boolean |
To enable masking or not. Note the masker is designed for SQL: on a Qlik load script it removes literal values but flattens the script to a single line |
false |
|
compress |
boolean |
To gzip the output or not |
true |
These parameters can be added directly into the run or you can pass the parameters in via a JSON file. The following is an example you can use that is included in the example run code below.
kada_qlik_extractor_config.json
{
"server": "",
"client_pfx": "",
"client_pfx_password": "",
"root_cer": "",
"user_directory": "",
"user_id": "",
"api_host": "",
"log_root": "",
"audit_root": "",
"audit_max_age_days": 90,
"stream_id": "",
"mapping": {},
"output_path": "C:/kada/output",
"mask": false,
"compress": true
}
Windows paths in the JSON can be written with forward slashes, e.g. "//qlikprd01/QlikShare/ArchivedLogs", or with every backslash doubled, e.g. "\\\\qlikprd01\\QlikShare\\ArchivedLogs".
Step 5: Run the Collector
The following code is an example of how to run the extractor. You may need to uplift this code to meet any code standards at your organisation.
This can be executed in any python environment where the whl has been installed.
This is the wrapper script: kada_qlik_extractor.py
import os
import argparse
from kada_collectors.extractors.utils import load_config, get_hwm, publish_hwm, get_generic_logger
from kada_collectors.extractors.qlik import Extractor
get_generic_logger('root') # Set to use the root logger, you can change the context accordingly or define your own logger
_type = 'qlik'
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'kada_{}_extractor_config.json'.format(_type))
parser = argparse.ArgumentParser(description='KADA Qlik Sense Extractor.')
parser.add_argument('--config', '-c', dest='config', default=filename, help='Location of the configuration json, default is the config json in the same directory as the script.')
parser.add_argument('--name', '-n', dest='name', default=_type, help='Name of the collector instance.')
args = parser.parse_args()
start_hwm, end_hwm = get_hwm(args.name)
ext = Extractor(**load_config(args.config))
ext.test_connection()
ext.run(**{"start_hwm": start_hwm, "end_hwm": end_hwm})
# ext.next_hwm, not end_hwm
# Qlik's reload history comes from an archived log SHARE, which is routinely behind
# the estate (the extractor reports this as "the log share is BEHIND"). Advancing
# the watermark to end_hwm would step over reloads that reach the share after this
# run, and they would never be read again. next_hwm is the newest event actually
# emitted, so a lagging archive is simply re-read next time.
publish_hwm(args.name, ext.next_hwm)
Important: publish ext.next_hwm, not end_hwm as the other collector wrappers do. The archived logs can arrive on the share later than the reloads and views they record, and publishing end_hwm would skip them permanently.
test_connection checks both the Qlik Repository (port 4242) and the Engine (port 4747), as they are often opened separately on the firewall.
The collector opens every app in scope in the Qlik Engine to read its data model. On a large estate, we recommend scheduling the collector outside business hours.
Advance options:
If you wish to maintain your own high water mark files elsewhere you can use the above section's script as a guide on how to call the extractor. The configuration file is simply the keyword arguments in JSON format. Refer to Collector Integration General Notes | Storing High Water Marks (HWM) for more information.
from kada_collectors.extractors.qlik import Extractor
kwargs = {} # However you choose to construct your args
hwm_kwargs = {"start_hwm": "YYYY-MM-DD HH:MM:SS", "end_hwm": "YYYY-MM-DD HH:MM:SS"} # UTC
ext = Extractor(**kwargs)
ext.run(**hwm_kwargs)
next_hwm = ext.next_hwm # Store this as the start_hwm of the next run
class Extractor(server: str = None, client_pfx: str = None, root_cer: str = None, \
user_directory: str = None, user_id: str = None, client_pfx_password: str = '', \
api_host: str = None, log_root: str = None, audit_root: str = '', stream_id: str = '', \
output_path: str = './output', compress: bool = False, mask: bool = False, \
mapping: dict = None, audit_max_age_days: int = 90) -> None
Step 6: Check the Collector Outputs
K Extracts
A set of files will be generated in the output_path directory:
|
File |
Contents |
|---|---|
|
META |
Streams, apps, sheets, data models, tables, fields, QVD files, load scripts and SQL queries, reloads |
|
LINKAGES |
Lineage between databases, files, QVDs, apps and sheets |
|
PROPERTIES |
Additional app and reload properties, e.g. reload task status and last reload time |
|
USERS |
Qlik users |
|
EVENTS |
Views and downloads (audit logs), reloads (reload logs), QVD writes, and app / sheet creation and modification |
Metadata and lineage are extracted in full on every run. Only events are limited to the period since the last run.
High Water Mark File
A high water mark file is created in the same directory as the execution called qlik_hwm.txt. This file is only produced if you call the publish_hwm method.
Step 7: Push the Extracts to K
Once the files have been validated, you can push the files to the K landing directory.
Known limitations
-
Lineage comes from the archived reload logs. An app's lineage reflects its most complete reload in the log archive. An app with no reload log in the archive has no lineage.
-
Apps using section access may refuse the service account in the Engine. Their tables and fields are then not collected, and their lineage comes from the reload log only.
-
Unpublished apps (in users' personal work areas) are not collected.
-
Apps not reloaded in 90 days keep their metadata and lineage, but are no longer shown as the producer of the QVDs they once wrote.
-
Date stamped QVD files (e.g.
sales_20260131.qvd,sales_20260201.qvd) are shown as a single QVD. -
Views and downloads history on the first run goes back 90 days by default. Set
audit_max_age_daysto read further back, as far as the audit logs are kept in the archive. -
ODBC DSN connections need a
mappingentry to join to the database source in K.
Troubleshooting
|
Message |
Cause / fix |
|---|---|
|
|
A required config field is empty |
|
|
The certificate path is wrong, or not readable by the account running the collector |
|
|
The ArchivedLogs path is wrong, or the Windows account running the collector has no access to the share |
|
|
|
|
|
Port 4242 is closed, |
|
|
The service account cannot see any apps (check it has the ContentAdmin role), or |
|
|
Port 4747 is closed, or the service account cannot open apps in the Engine |
|
|
That app could not be opened in the Engine, usually because of section access. Its tables and fields are skipped, lineage still comes from its reload log |
|
QVDs reported as |
The Windows account running the collector has no read access to that share |
|
|
An app connects through an ODBC DSN or connection with no host. Add it to |