Install KCM
KCM is self-hosted software: it runs on your own machines, next to your Kafka clusters. Pick Docker Compose for a workstation or a single server, or the Helm chart for Kubernetes.
Architecture
A KCM installation has four parts:
| Component | Role |
|---|---|
| kcm-ui | The web console, served as static files. |
| kcm-api | The REST API (/api/v1) and the WebSocket endpoint
(/ws/v1) used by live views. It is the only component that talks to Kafka, Kafka Connect,
Schema Registry, ksqlDB and Confluent Cloud. |
| PostgreSQL | Stores the configuration: clusters, users, roles, audit trail, templates, test scenarios. Secrets are encrypted at rest. |
| Redis | Cache shared by the API instances. |
From 1.1.0 on, kcm-api and kcm-ui share one version number: always deploy the same
version of both. The Helm chart version is the KCM version it deploys (chart 1.5.1 deploys the
1.5.1 images).
Docker Compose
The public repository github.com/kcmhub/KCM holds a ready to use Compose file with PostgreSQL and Redis.
Requirements
- Docker Engine with the Compose plugin;
- ports
80(console) and8080(API) free on the host; - network access from the host to your Kafka brokers.
Start KCM
- Clone the repository and open the Compose folder:
git clone https://github.com/kcmhub/KCM.git cd KCM/00_kcm_with_docker_compose - Choose the password of the built-in
adminuser before the first start (otherwise change it right after signing in):export KCM_DEFAULT_PASSWORD='choose-a-strong-password' - Create the Docker network and start the stack:
docker network create kcm-net docker compose up -d - Open
http://localhostand sign in asadmin. KCM asks for a new password at the first sign in.
Key pairs
At the first start the API generates two key pairs in data/keys:
data/keys/jwt/signs the sign in sessions;data/keys/encryption/encrypts the secrets KCM stores in its database (SASL passwords, truststore passwords, Schema Registry, Kafka Connect, ksqlDB, SSO, Git and Confluent Cloud credentials).
Without the encryption key pair, the secrets stored in the database cannot be read and must be entered again. Never share this folder. See Upgrades and backups.
To configure the stack, edit the environment section of the kcm-api service. Every
variable is described in the configuration reference.
Upgrade a Compose install
cd KCM && git pull
cd 00_kcm_with_docker_compose
docker compose pull && docker compose up -d
The database schema is updated at startup. Read the upgrade notes before jumping several versions, in particular from 1.2.0 or earlier.
Kubernetes with Helm
The chart is a public OCI artifact on Docker Hub, next to the images: no registry login is needed.
Requirements
- Kubernetes 1.22 or later;
- Helm 3.8 or later (OCI registries);
- an Ingress controller, for example ingress-nginx;
- a default StorageClass, or
storageClassset in the values, for the volumes; - the Secret holding the JWT key pair, created before the install.
Install
- Create the namespace and the JWT key pair of the API:
kubectl create namespace kcm mkdir -p keys openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out keys/private.pem openssl pkey -in keys/private.pem -pubout -out keys/public.pem kubectl -n kcm create secret generic kcm-api-keys --from-file=keys - Recommended for a new install: a distinct encryption key pair, so that rotating the JWT pair never touches
the stored secrets:
mkdir -p encryption-keys openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out encryption-keys/private.pem openssl pkey -in encryption-keys/private.pem -pubout -out encryption-keys/public.pem kubectl -n kcm create secret generic kcm-api-encryption-keys --from-file=encryption-keys - Install the chart. This evaluation example uses the embedded PostgreSQL and Redis:
helm upgrade --install kcm oci://registry-1.docker.io/kafkaetech/kcm --version 1.5.1 -n kcm \ --set ingress.host=kcm.example.com \ --set ui.runtimeConfig.apiUrl=https://kcm.example.com \ --set ui.runtimeConfig.wsUrl=wss://kcm.example.com \ --set api.encryptionKeys.enabled=true \ --set postgres.enabled=true --set redis.enabled=true - Read the generated
adminpassword (the command is also printed byhelm install), then sign in and change it:kubectl -n kcm get secret kcm-api-secret -o jsonpath="{.data.KCM_DEFAULT_PASSWORD}" | base64 -d
Read every value of the chart with:
helm show values oci://registry-1.docker.io/kafkaetech/kcm --version 1.5.1
Typical production values
# values.yaml
ingress:
enabled: true
className: nginx
host: kcm.example.com
tls:
enabled: true
secretName: kcm-tls # empty: certificate of the Ingress controller
ui:
runtimeConfig:
apiUrl: https://kcm.example.com
wsUrl: wss://kcm.example.com
api:
keys:
existingSecret: kcm-api-keys
encryptionKeys:
enabled: true # new installs only, see the upgrade notes
existingSecret: kcm-api-encryption-keys
env:
KCM_DB_URL: jdbc:postgresql://postgres.example.internal:5432/kcm_db
KCM_DB_USER: kcm_user
KCM_REDIS_HOST: redis.example.internal
secret:
KCM_DB_PWD: "" # required with an external PostgreSQL, pass it with --set or a sealed value
postgres:
enabled: false
redis:
enabled: false
helm upgrade --install kcm oci://registry-1.docker.io/kafkaetech/kcm --version 1.5.1 \
-n kcm -f values.yaml --set api.secret.KCM_DB_PWD="$KCM_DB_PWD"
Environment variables of the API go under api.env, passwords under api.secret. The
full list is in the configuration reference.
Passwords
The chart ships no default password:
| Value | When empty |
|---|---|
api.secret.KCM_DEFAULT_PASSWORD | Password of the admin user, generated
at install. Used only when KCM creates the user at its first start. |
postgres.auth.password | Password of the embedded PostgreSQL, generated at install. |
api.secret.KCM_DB_PWD | Password of an external PostgreSQL: required, the install fails without it. |
On helm upgrade, an empty value keeps the password of the deployed Secret. This relies on Helm
lookup, which returns nothing with helm template, Argo CD or a client-side
--dry-run: there, set the passwords explicitly.
Volumes
The API mounts the JWT key pair (read only, from kcm-api-keys), the encryption key pair when
api.encryptionKeys.enabled=true, a files volume (truststores, keytabs and other uploaded files), and
a license volume. With ReadWriteOnce volumes, keep api.replicas: 1 unless your storage
supports ReadWriteMany.
Ingress and reverse proxy
The console and the API must be served from the URLs set in ui.runtimeConfig. The chart routes:
| Path | Goes to |
|---|---|
/ | kcm-ui |
/api/ | kcm-api (REST API) |
/auth/sso | kcm-api (start of an SSO sign in) |
/oauth2/, /login/oauth2/ | kcm-api (OIDC sign in and its return URL), chart 1.5.1 or later |
/saml2/, /login/saml2/, /logout/saml2/ | kcm-api (SAML sign in, return URL and sign out), chart 1.5.1 or later |
/ws/v1 | kcm-api (WebSocket of the live views) |
/actuator | kcm-api; only /actuator/health answers without signing
in |
The WebSocket needs HTTP/1.1 upgrade and long timeouts. With ingress-nginx, the chart sets these annotations by default:
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
SSO behind the Ingress of the chart needs chart 1.5.1 or
later: older charts do not route the SSO return paths to the API. Behind your own Ingress or another
reverse proxy, forward all the paths of the table to the API and allow WebSocket upgrades on /ws/v1. Set
KCM_REDIRECT_ALLOWLIST to the public origin of the console (the Helm chart does it from the Ingress
host): the live message consumer only accepts the browser origins listed there.
First steps after the install
- Sign in as
adminand set a new password when asked. - Turn on two-factor sign in for the administrator.
- Install your license, or request the 30-day Enterprise trial, see Editions and licensing.
- Create a KCM group and register your first Kafka cluster.
- If your brokers or registries use a private certificate authority, add it in Settings > Trusted certificates.
- Create users or connect your identity provider, then grant roles, see Roles and access control.