KCM as code

Declare who can do what in KCM in YAML files kept in Git, review the changes in a merge request, and let your pipeline apply them. KCM shows the plan before anything changes and keeps an audit record of every apply.

Since KCM 1.5.0 Every edition REST API: /api/v1/iac

What it is and why

KCM as code manages the access control of KCM from declarations instead of clicks. In 1.5.0 it covers three kinds of objects:

Why manage access this way:

Secrets never go through the files

Declarations hold names, roles and scopes only. Service account tokens are created in the console or with POST /api/v1/users/service-accounts/{name}/tokens, see Service accounts and tokens.

Editions and permissions

KCM as code is available in every edition CommunityStarterTeamEnterprise. The objects you declare follow the rules of your edition:

Who can call the API:

ActionGlobal role needed
Export, list managed objectsADMIN, SECURITY_ADMIN or AUDITOR
Plan, applyADMIN or SECURITY_ADMIN
Release an object from managementADMIN

A SECURITY_ADMIN (Enterprise) cannot grant anything to itself, nor to a role group it belongs to. See Roles and access control.

The YAML format

A declaration set is one of:

Every document has the same envelope:

apiVersion: kcm.kcmhub.io/v1
kind: RoleGroup            # RoleGroup, ServiceAccount or RoleBinding
metadata:
  name: payments-team      # identifies the object within its kind
spec:
  ...

metadata.name accepts letters, digits and the characters . _ @ : -. A RoleGroup is named after the role group, a ServiceAccount after the service account. A RoleBinding name is yours to choose: it only identifies the binding in the files and in the plan.

RoleGroup

apiVersion: kcm.kcmhub.io/v1
kind: RoleGroup
metadata:
  name: payments-team
spec:
  description: Payments squad
  enabled: true
  members: [alice@example.com, bob@example.com]
FieldDescription
descriptionOptional text.
enabledtrue by default. A disabled role group grants nothing.
membersUsernames of existing users (local or SSO). Users are never created by the declarations. Memberships that come from your identity provider at SSO sign in are not listed here.

ServiceAccount

apiVersion: kcm.kcmhub.io/v1
kind: ServiceAccount
metadata:
  name: payments-ci
spec:
  description: Deployment pipeline of the payments services
  enabled: true

description is optional; enabled is true by default. Disabling a service account stops its tokens from working and frees its seat.

RoleBinding

A role binding grants either a role on a resource, or a list of operations on resources matching a pattern.

# A role on a Kafka cluster
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
  name: payments-team-editor-prod
spec:
  subject: { kind: RoleGroup, name: payments-team }
  role: editor                       # viewer, editor or admin
  scope: /groups/prod/clusters/payments
---
# Operations on the topics starting with "payments."
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
  name: payments-ci-topics
spec:
  subject: { kind: ServiceAccount, name: payments-ci }
  operations: [READ, WRITE, DESCRIBE]
  patternType: PREFIXED              # LITERAL (default) or PREFIXED
  scope: /groups/prod/clusters/payments/topics/payments.
  description: CI reads and writes the payments topics
FieldDescription
subjectkind and name of who receives the grant, see below.
roleviewer, editor or admin, for a scope that ends on a resource (KCM group, Kafka cluster, Kafka Connect cluster, ksqlDB endpoint).
operationsFor a scope that ends on a pattern (topics, consumer groups, subjects, connectors). Allowed values depend on the resource, see Scopes.
patternTypeLITERAL (default, exact name) or PREFIXED (every name starting with the pattern). Pattern grants only.
scopePath of the resource, by names.
enabledtrue by default.
descriptionOptional text.

Subjects

subject.kindsubject.name
UserAn existing user, local or SSO. Users are not created by the declarations.
ServiceAccountA service account, existing or declared in the same set.
RoleGroupA role group, existing or declared in the same set.

Scopes

A scope addresses a resource by its names, from the KCM group down. Grants add up: a role on a KCM group applies to its clusters, a role on a Kafka cluster gives every topic, consumer group, subject and connector of it, and pattern grants add resources to users who have no role there.

ScopeGrants
/groups/{group}Role on a KCM group and everything below it.
/groups/{group}/clusters/{cluster}Role on a Kafka cluster, its Kafka Connect clusters and ksqlDB endpoints.
.../clusters/{cluster}/connect/{connect}Role on a Kafka Connect cluster.
.../clusters/{cluster}/ksql/{endpoint}Role on a ksqlDB endpoint (users and service accounts only, not role groups).
.../clusters/{cluster}/topics/{pattern}Topic operations: READ, WRITE, CREATE, DELETE, ALTER, DESCRIBE, ALL.
.../clusters/{cluster}/consumer-groups/{pattern}Consumer group operations: DESCRIBE, DELETE, RESET_OFFSETS, ALL.
.../clusters/{cluster}/schema-registries/{registry}/subjects/{pattern}Subject operations: DESCRIBE, CREATE, DELETE, ALTER, ALL.
.../clusters/{cluster}/connect/{connect}/connectors/{pattern}Connector operations: DESCRIBE, UPSERT, DELETE, CONTROL, ALL.

The connector CONTROL operation covers pause, resume, restart and stop only. Updating or deleting a connector needs UPSERT or DELETE.

Names with spaces, slashes or percent signs

Each name of the path (KCM group, cluster, Kafka Connect cluster, Schema Registry, ksqlDB endpoint) is one segment and is percent-decoded:

# KCM group "Data Platform", cluster "eu/west 1"
scope: /groups/Data%20Platform/clusters/eu%2Fwest%201
# the same group, space written as is
scope: /groups/Data Platform/clusters/eu%2Fwest%201

The pattern at the end of a topic, consumer group, subject or connector scope is taken as is, without decoding: .../topics/orders/v1 is the topic pattern orders/v1.

The export writes names with %2F and %25 only, so files exported from any 1.5.x release keep their meaning.

Unique names

Kafka Connect clusters, Schema Registries and ksqlDB endpoints must have unique names within their Kafka cluster, so that a scope always points to one resource. At startup, KCM 1.5.0 renames the duplicates left by older versions (name-2, name-3...), logs and audits each rename, and it refuses a duplicate name when one is created or renamed.

Export, plan, apply

The usual cycle is:

  1. Export what KCM holds today, to start your repository from the real state.
  2. Edit the files and open a merge request.
  3. Plan: KCM compares the files with its state and lists the changes. Nothing changes.
  4. Apply after review: KCM makes the changes in one transaction.

The source (managedBy)

Plan and apply take a source parameter that names the declarations, for example a repository or a pipeline: git:payments-access. It accepts letters, digits and . _ : / @ -. Every object applied is marked managedBy: iac:{source}.

Export

GET /api/v1/iac/export?kinds=RoleGroup,ServiceAccount,RoleBinding&managedBy=iac:{source}&format=yaml

Managed objects keep their declared name. Objects not managed yet get a stable generated name, so exporting twice gives the same file.

curl -s -H "Authorization: Bearer $KCM_TOKEN" \
  "https://kcm.example.com/api/v1/iac/export?format=yaml" > access.yaml

Plan

POST /api/v1/iac/plan?source={source}&prune=true|false

Send the declarations as the request body (Content-Type: application/yaml or application/json). The answer lists each change and every error:

{
  "managedBy": "iac:git:payments-access",
  "applied": false,
  "changes": [
    { "action": "CREATE", "kind": "RoleGroup",  "name": "payments-team", "detail": "2 member(s)" },
    { "action": "ADOPT",  "kind": "ServiceAccount", "name": "payments-ci", "detail": null },
    { "action": "NOOP",   "kind": "RoleBinding", "name": "payments-ci-topics", "detail": null }
  ],
  "errors": []
}
ActionMeaning
CREATEThe object does not exist and will be created.
UPDATEThe object exists and differs from the declaration.
DELETEOnly with prune=true: an object of this source is no longer declared.
ADOPTAn existing object matches the declaration: it becomes managed without any change.
NOOPAlready managed and identical.

Errors (unknown user, unknown cluster in a scope, operation not allowed for a resource, object owned by another source, edition limits...) are listed in errors. A plan with errors cannot be applied.

Apply

POST /api/v1/iac/apply?source={source}&prune=true|false

Same body and same answer as the plan, with "applied": true. All changes are made in one transaction: either every change is applied, or none. The call is refused with 400 when the plan has errors.

Prune

With prune=true, objects of this source that are no longer declared are removed:

Without prune (the default), removing a document from the files leaves the object in KCM, still managed by the source. Run the plan with prune=true first to see what would go.

Managed objects are read only

Once applied, a managed role group, service account or role binding can only be changed through its declarations:

A grant added in the console to a managed role group is not managed: the next plan with prune leaves it alone, because only the objects of the source are pruned.

Drift

Any difference between the files and KCM shows in the next plan as an UPDATE, CREATE or DELETE. To check drift on a schedule, run a plan and fail the job when it lists anything other than NOOP.

Release an object

To hand an object back to the console, list the managed objects and remove the marking (global ADMIN). The object is kept as is, becomes editable again and is no longer pruned by its source.

# Managed objects, optionally filtered by kind
curl -s -H "Authorization: Bearer $KCM_TOKEN" "https://kcm.example.com/api/v1/iac/managed?kind=RoleBinding"

# Release one object
curl -s -X DELETE -H "Authorization: Bearer $KCM_TOKEN" \
  "https://kcm.example.com/api/v1/iac/managed/RoleBinding/payments-ci-topics"

Examples with curl

The examples use a token in $KCM_TOKEN whose account has the global ADMIN or SECURITY_ADMIN role:

KCM_URL=https://kcm.example.com
SOURCE=git:payments-access

# 1. Plan, without and with prune
curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
  --data-binary @access.yaml "$KCM_URL/api/v1/iac/plan?source=$SOURCE&prune=true"

# 2. Apply
curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
  --data-binary @access.yaml "$KCM_URL/api/v1/iac/apply?source=$SOURCE&prune=true"

# 3. Export what this source manages, to compare with the repository
curl -sf -H "Authorization: Bearer $KCM_TOKEN" \
  "$KCM_URL/api/v1/iac/export?managedBy=iac:$SOURCE&format=yaml"

Use --data-binary rather than -d, so the line breaks of the YAML file are kept.

CI example: plan on merge request, apply on main

A typical repository holds one or more YAML files (here access/*.yaml). The pipeline plans on every merge request and applies when the change reaches the main branch. Store the token as a masked, protected CI variable.

GitLab CI

# .gitlab-ci.yml
variables:
  KCM_URL: https://kcm.example.com
  KCM_SOURCE: git:platform-access
  # KCM_TOKEN: masked and protected CI/CD variable

.kcm:
  image: alpine:3.20
  before_script:
    - apk add --no-cache curl jq
    # one stream of YAML documents: a --- line between two files
    - awk 'FNR==1 && NR!=1 {print "---"} {print}' access/*.yaml > all.yaml

kcm-plan:
  extends: .kcm
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - |
      curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
        --data-binary @all.yaml "$KCM_URL/api/v1/iac/plan?source=$KCM_SOURCE&prune=true" | tee plan.json
    - jq -r '.changes[] | select(.action != "NOOP") | "\(.action)\t\(.kind)\t\(.name)\t\(.detail)"' plan.json
    - test "$(jq '.errors | length' plan.json)" -eq 0 || (jq -r '.errors[]' plan.json; exit 1)
  artifacts:
    paths: [plan.json]

kcm-apply:
  extends: .kcm
  stage: deploy
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  resource_group: kcm-access
  script:
    - |
      curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
        --data-binary @all.yaml "$KCM_URL/api/v1/iac/apply?source=$KCM_SOURCE&prune=true" | tee apply.json
    - jq -e '.applied' apply.json

GitHub Actions

# .github/workflows/kcm-access.yml
name: kcm-access
on:
  pull_request:
    paths: ["access/**"]
  push:
    branches: [main]
    paths: ["access/**"]

env:
  KCM_URL: https://kcm.example.com
  KCM_SOURCE: git:platform-access

jobs:
  kcm:
    runs-on: ubuntu-latest
    concurrency: kcm-access
    steps:
      - uses: actions/checkout@v4
      - name: Merge the YAML files
        run: awk 'FNR==1 && NR!=1 {print "---"} {print}' access/*.yaml > all.yaml
      - name: Plan
        if: github.event_name == 'pull_request'
        env:
          KCM_TOKEN: ${{ secrets.KCM_TOKEN }}
        run: |
          curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
            --data-binary @all.yaml "$KCM_URL/api/v1/iac/plan?source=$KCM_SOURCE&prune=true" > plan.json
          jq -r '.changes[] | select(.action != "NOOP") | "\(.action) \(.kind) \(.name)"' plan.json
          test "$(jq '.errors | length' plan.json)" -eq 0 || { jq -r '.errors[]' plan.json; exit 1; }
      - name: Apply
        if: github.event_name == 'push'
        env:
          KCM_TOKEN: ${{ secrets.KCM_TOKEN }}
        run: |
          curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
            --data-binary @all.yaml "$KCM_URL/api/v1/iac/apply?source=$KCM_SOURCE&prune=true" | jq -e '.applied'
Good practice
  • Give the pipeline its own service account, with the global SECURITY_ADMIN role on Enterprise, or ADMIN otherwise, and a token with a validity period.
  • Run one apply at a time (resource_group, concurrency).
  • Start without prune, check a few plans, then turn it on.
  • Do not declare the service account that runs the pipeline in the set it applies (create it in the console, or declare it under another source), so a mistake in the files cannot disable it.

Audit of the changes

Every call is recorded in the audit trail:

ActionRecorded
IAC_APPLY_REQUESTEach apply call, with the source and whether prune was set.
IAC_APPLYEach successful apply, with the source and the number of changes by kind of change.
IAC_UNMANAGEAn object released from management, with its kind and name.

The changes themselves (role groups, members, grants, service accounts) are also recorded as usual, attributed to the account that called the API. On Enterprise, these events can be exported to CSV and streamed to your SIEM.

Complete example

# access/payments.yaml
apiVersion: kcm.kcmhub.io/v1
kind: RoleGroup
metadata:
  name: payments-team
spec:
  description: Payments squad
  members: [alice@example.com, bob@example.com]
---
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
  name: payments-team-viewer-prod
spec:
  subject: { kind: RoleGroup, name: payments-team }
  role: viewer
  scope: /groups/Production/clusters/payments
---
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
  name: payments-team-editor-staging
spec:
  subject: { kind: RoleGroup, name: payments-team }
  role: editor
  scope: /groups/Staging/clusters/payments
---
apiVersion: kcm.kcmhub.io/v1
kind: ServiceAccount
metadata:
  name: payments-ci
spec:
  description: Deployment pipeline of the payments services
---
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
  name: payments-ci-topics
spec:
  subject: { kind: ServiceAccount, name: payments-ci }
  operations: [READ, WRITE, DESCRIBE, CREATE]
  patternType: PREFIXED
  scope: /groups/Production/clusters/payments/topics/payments.
---
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
  name: payments-ci-groups
spec:
  subject: { kind: ServiceAccount, name: payments-ci }
  operations: [DESCRIBE, RESET_OFFSETS]
  patternType: PREFIXED
  scope: /groups/Production/clusters/payments/consumer-groups/payments-

Coming next

Planned, not available in 1.5.0

KCM as code will grow beyond access control: more resource kinds, and a Terraform provider built on the same declarations. This section will describe them when they are released. Until then, only RoleGroup, ServiceAccount and RoleBinding are accepted.