Install KCM

KCM is self-hosted software: it runs on your own machines, next to your Kafka clusters. Pick Docker Compose for a workstation or a single server, or the Helm chart for Kubernetes.

Images: kafkaetech/kcm-api and kafkaetech/kcm-ui Helm: oci://registry-1.docker.io/kafkaetech/kcm

Architecture

A KCM installation has four parts:

ComponentRole
kcm-uiThe web console, served as static files.
kcm-apiThe REST API (/api/v1) and the WebSocket endpoint (/ws/v1) used by live views. It is the only component that talks to Kafka, Kafka Connect, Schema Registry, ksqlDB and Confluent Cloud.
PostgreSQLStores the configuration: clusters, users, roles, audit trail, templates, test scenarios. Secrets are encrypted at rest.
RedisCache shared by the API instances.

From 1.1.0 on, kcm-api and kcm-ui share one version number: always deploy the same version of both. The Helm chart version is the KCM version it deploys (chart 1.5.1 deploys the 1.5.1 images).

Docker Compose

The public repository github.com/kcmhub/KCM holds a ready to use Compose file with PostgreSQL and Redis.

Requirements

Start KCM

  1. Clone the repository and open the Compose folder:
    git clone https://github.com/kcmhub/KCM.git
    cd KCM/00_kcm_with_docker_compose
  2. Choose the password of the built-in admin user before the first start (otherwise change it right after signing in):
    export KCM_DEFAULT_PASSWORD='choose-a-strong-password'
  3. Create the Docker network and start the stack:
    docker network create kcm-net
    docker compose up -d
  4. Open http://localhost and sign in as admin. KCM asks for a new password at the first sign in.

Key pairs

At the first start the API generates two key pairs in data/keys:

Back up data/keys with the database

Without the encryption key pair, the secrets stored in the database cannot be read and must be entered again. Never share this folder. See Upgrades and backups.

To configure the stack, edit the environment section of the kcm-api service. Every variable is described in the configuration reference.

Upgrade a Compose install

cd KCM && git pull
cd 00_kcm_with_docker_compose
docker compose pull && docker compose up -d

The database schema is updated at startup. Read the upgrade notes before jumping several versions, in particular from 1.2.0 or earlier.

Kubernetes with Helm

The chart is a public OCI artifact on Docker Hub, next to the images: no registry login is needed.

Requirements

Install

  1. Create the namespace and the JWT key pair of the API:
    kubectl create namespace kcm
    mkdir -p keys
    openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out keys/private.pem
    openssl pkey -in keys/private.pem -pubout -out keys/public.pem
    kubectl -n kcm create secret generic kcm-api-keys --from-file=keys
  2. Recommended for a new install: a distinct encryption key pair, so that rotating the JWT pair never touches the stored secrets:
    mkdir -p encryption-keys
    openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out encryption-keys/private.pem
    openssl pkey -in encryption-keys/private.pem -pubout -out encryption-keys/public.pem
    kubectl -n kcm create secret generic kcm-api-encryption-keys --from-file=encryption-keys
  3. Install the chart. This evaluation example uses the embedded PostgreSQL and Redis:
    helm upgrade --install kcm oci://registry-1.docker.io/kafkaetech/kcm --version 1.5.1 -n kcm \
      --set ingress.host=kcm.example.com \
      --set ui.runtimeConfig.apiUrl=https://kcm.example.com \
      --set ui.runtimeConfig.wsUrl=wss://kcm.example.com \
      --set api.encryptionKeys.enabled=true \
      --set postgres.enabled=true --set redis.enabled=true
  4. Read the generated admin password (the command is also printed by helm install), then sign in and change it:
    kubectl -n kcm get secret kcm-api-secret -o jsonpath="{.data.KCM_DEFAULT_PASSWORD}" | base64 -d

Read every value of the chart with:

helm show values oci://registry-1.docker.io/kafkaetech/kcm --version 1.5.1

Typical production values

# values.yaml
ingress:
  enabled: true
  className: nginx
  host: kcm.example.com
  tls:
    enabled: true
    secretName: kcm-tls            # empty: certificate of the Ingress controller

ui:
  runtimeConfig:
    apiUrl: https://kcm.example.com
    wsUrl: wss://kcm.example.com

api:
  keys:
    existingSecret: kcm-api-keys
  encryptionKeys:
    enabled: true                  # new installs only, see the upgrade notes
    existingSecret: kcm-api-encryption-keys
  env:
    KCM_DB_URL: jdbc:postgresql://postgres.example.internal:5432/kcm_db
    KCM_DB_USER: kcm_user
    KCM_REDIS_HOST: redis.example.internal
  secret:
    KCM_DB_PWD: ""                 # required with an external PostgreSQL, pass it with --set or a sealed value

postgres:
  enabled: false
redis:
  enabled: false
helm upgrade --install kcm oci://registry-1.docker.io/kafkaetech/kcm --version 1.5.1 \
  -n kcm -f values.yaml --set api.secret.KCM_DB_PWD="$KCM_DB_PWD"

Environment variables of the API go under api.env, passwords under api.secret. The full list is in the configuration reference.

Passwords

The chart ships no default password:

ValueWhen empty
api.secret.KCM_DEFAULT_PASSWORDPassword of the admin user, generated at install. Used only when KCM creates the user at its first start.
postgres.auth.passwordPassword of the embedded PostgreSQL, generated at install.
api.secret.KCM_DB_PWDPassword of an external PostgreSQL: required, the install fails without it.

On helm upgrade, an empty value keeps the password of the deployed Secret. This relies on Helm lookup, which returns nothing with helm template, Argo CD or a client-side --dry-run: there, set the passwords explicitly.

Volumes

The API mounts the JWT key pair (read only, from kcm-api-keys), the encryption key pair when api.encryptionKeys.enabled=true, a files volume (truststores, keytabs and other uploaded files), and a license volume. With ReadWriteOnce volumes, keep api.replicas: 1 unless your storage supports ReadWriteMany.

Ingress and reverse proxy

The console and the API must be served from the URLs set in ui.runtimeConfig. The chart routes:

PathGoes to
/kcm-ui
/api/kcm-api (REST API)
/auth/ssokcm-api (start of an SSO sign in)
/oauth2/, /login/oauth2/kcm-api (OIDC sign in and its return URL), chart 1.5.1 or later
/saml2/, /login/saml2/, /logout/saml2/kcm-api (SAML sign in, return URL and sign out), chart 1.5.1 or later
/ws/v1kcm-api (WebSocket of the live views)
/actuatorkcm-api; only /actuator/health answers without signing in

The WebSocket needs HTTP/1.1 upgrade and long timeouts. With ingress-nginx, the chart sets these annotations by default:

nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
nginx.ingress.kubernetes.io/proxy-buffering: "off"

SSO behind the Ingress of the chart needs chart 1.5.1 or later: older charts do not route the SSO return paths to the API. Behind your own Ingress or another reverse proxy, forward all the paths of the table to the API and allow WebSocket upgrades on /ws/v1. Set KCM_REDIRECT_ALLOWLIST to the public origin of the console (the Helm chart does it from the Ingress host): the live message consumer only accepts the browser origins listed there.

First steps after the install

  1. Sign in as admin and set a new password when asked.
  2. Turn on two-factor sign in for the administrator.
  3. Install your license, or request the 30-day Enterprise trial, see Editions and licensing.
  4. Create a KCM group and register your first Kafka cluster.
  5. If your brokers or registries use a private certificate authority, add it in Settings > Trusted certificates.
  6. Create users or connect your identity provider, then grant roles, see Roles and access control.