1. Deploying KDQ
KDQ can be deployed on “your cloud” infrastructure that runs as three workloads in a single Kubernetes namespace (kdq):
|
Workload |
Role |
|
agent-api |
Stateless web/API service behind your ingress. Autoscales. |
|
agent-worker |
Background engine for profiling, rule execution, and scheduled exports. Single instance. |
|
agent-db |
PostgreSQL 18 database, running inside the cluster on a persistent disk. |
KDQ requires a database. KDQ can either be deployed in cluster or use a managed PostgresSQL service (e.g. Azure Database).
Note: Use the prod-pg-in-cluster overlay for in cluster deployment. PostgreSQL runs in-cluster on a persistent volume that you own and back up. With in-cluster PostgreSQL, the pgdata disk is the only copy of all KDQ state — sources, rules, run history, and encrypted credentials. There is no managed provider taking backups or handling failover. Provision durable SSD storage and configure backups before putting real data in it.
1.1 Cloud deployment options
KDQ is deployed on Kubernetes. The following Kubernetes providers are recommended
|
Requirement |
Detail |
|
AWS |
Elastic Kubernetes Services (EKS) |
|
Azure |
Azure Kubernete Services (AKS) |
|
Google Cloud |
Google Kubernetes Engine (GKE) |
|
Redhat |
Openshift |
|
SUSE |
Rancher |
2. Kubernetes cluster requirements
|
Requirement |
Detail |
|
Kubernetes version |
1.25 or later recommended. The manifests use only GA APIs (apps/v1, autoscaling/v2, policy/v1, networking.k8s.io/v1). |
|
Cluster access |
kubectl with cluster-admin on the target cluster (creates a namespace and namespaced objects). |
|
Ingress controller |
An ingress controller must be installed — ingress-nginx is assumed by the overlay's annotations. (Traefik Gateway API is supported as an alternative; see the install runbook.) |
|
Metrics server |
metrics-server must be running. The API autoscaler (HPA) scales on CPU utilisation and will not function without it. |
|
StorageClass |
A dynamic, SSD-backed StorageClass for the database disk (e.g. AKS managed-csi-premium, AWS gp3, GCP pd-ssd). Spinning disk will hurt PostgreSQL. |
|
Worker nodes |
At least 2 worker nodes recommended, so the API stays available (PodDisruptionBudget keeps ≥1 API pod) during node drains and rollouts. |
3. Compute & memory sizing
These are the per-workload requests and limits the overlay ships with (reasoned starting points — monitor and adjust):
|
Workload |
Replicas |
CPU request |
Mem request |
CPU limit |
Mem limit |
|
agent-api |
2–6 (autoscaled) |
250m |
512 Mi |
1 |
2 Gi |
|
agent-worker |
1 (fixed) |
500m |
1 Gi |
2 |
4 Gi |
|
agent-db |
1 (fixed) |
1 |
2 Gi |
2 |
4 Gi |
Cluster capacity to plan for (sum across all pods):
|
Scenario |
CPU |
Memory |
|
Minimum to schedule (sum of requests, API at floor of 2) |
~2 vCPU |
~4 GiB |
|
Recommended allocatable (room to burst + scale) |
8–12 vCPU |
16–24 GiB |
Scaling rules:
-
agent-api scales horizontally — the HPA runs 2→6 replicas at 70% CPU. Add API capacity by letting it scale, not by resizing pods.
-
agent-worker is a hard singleton — never add replicas. To give it more capacity, raise its CPU/memory. Worker memory is the first thing to raise if profiling large tables.
-
agent-db scales vertically only — raise its CPU/memory; there is no read-replica strategy.
4. Storage sizing
|
Volume |
Size |
Type |
Notes |
|
pgdata (database) |
20 GiB to start; plan 20–50 GiB |
SSD (required) |
The single source of truth for all KDQ data. Grows with profiling stats, audit log, and run history. |
-
The disk is provisioned automatically from your SSD StorageClass via a 20 GiB ReadWriteOnce PersistentVolumeClaim. Pin the SSD class before the first deploy — the database volume size and class are fixed once created.
-
No separate output volume is required. Rule-run output and catalogue exports are written to Azure Blob Storage (see Step 5), not to a cluster disk.
-
Each node also needs local disk for the container image and a small per-pod scratch area.
Note: Schedule logical dumps (pg_dump) to off-cluster storage, or volume snapshots of the pgdata disk if your storage driver supports them. Validate a restore at least once.
5. External services & credentials
Provision these before deploying — the platform expects them at startup:
|
Prerequisite |
Purpose |
Required? |
|
Container registry |
Holds the pinned KDQ image, pullable by the cluster. An image-pull secret (kada-image-credentials) must exist in the kdq namespace. |
Yes |
|
DNS record |
An A/CNAME for your KDQ hostname → the ingress load-balancer IP. |
Yes |
|
TLS certificate |
For that hostname. TLS terminates at the ingress (cert-manager or a cert you supply); KDQ never terminates TLS itself. |
Yes |
|
Microsoft Entra ID app registration |
OIDC single sign-on. Register the exact HTTPS redirect URI (https://<your-host>/auth/callback). |
Yes |
|
Azure Blob Storage |
A container plus a SAS URL — destination for all rule-run output and catalogue exports. |
Yes |
|
LLM provider key |
Anthropic / OpenAI / private Azure OpenAI key for AI rule authoring. Omit to run without AI features. |
Optional |
|
SMTP relay |
Email alert delivery. |
Optional |
You will also generate a one-time encryption key (a 64-character hex string) during install. It encrypts all stored source credentials — back it up out-of-band and never rotate it via redeploy, or every stored credential becomes unrecoverable.
6. Network requirements
Inbound: only the ingress controller needs to reach agent-api (HTTP, port 8400 via the service on port 80). No inbound access is required to the worker or the database.
Outbound (egress) from the agent-api and agent-worker pods — allow through your firewall:
|
Destination |
For |
|
Entra ID OIDC sign-in |
|
|
*.blob.core.windows.net |
Writing run output / exports to Azure Blob |
|
Your LLM provider endpoint |
AI rule authoring (if enabled) — e.g. api.anthropic.com, api.openai.com, or your Azure OpenAI host |
|
Your SMTP relay host |
Email alerts (if enabled) |
|
Your registered source database hosts |
Profiling and rule execution against the data being checked |
|
Your container registry |
Image pulls (from the cluster nodes) |