KCM as code
Declare who can do what in KCM in YAML files kept in Git, review the changes in a merge request, and let your pipeline apply them. KCM shows the plan before anything changes and keeps an audit record of every apply.
What it is and why
KCM as code manages the access control of KCM from declarations instead of clicks. In 1.5.0 it covers three kinds of objects:
- RoleGroup: a role group and its members;
- ServiceAccount: a non-human account used by scripts and pipelines;
- RoleBinding: a role or a set of operations granted to a user, a service account or a role group, on a scope (a KCM group, a Kafka cluster, a topic prefix...).
Why manage access this way:
- Review before change. Access changes go through a merge request, a reviewer reads the diff, and the pipeline shows the KCM plan before anything is applied.
- One reference. Applied objects are marked with their source and become read only in the console and the API, so the files stay the truth.
- Repeatable environments. The same files can be applied to a staging and a production KCM.
- Drift is visible. A change made outside the files shows in the next plan.
- Traceability. Every apply is recorded in the audit trail with the number of changes by kind.
Declarations hold names, roles and scopes only. Service account tokens are created in the console or with
POST /api/v1/users/service-accounts/{name}/tokens, see
Service accounts and tokens.
Editions and permissions
KCM as code is available in every edition CommunityStarterTeamEnterprise. The objects you declare follow the rules of your edition:
- RoleGroup and RoleBinding: every edition.
- ServiceAccount: service accounts exist from the Team edition on, and each enabled service account takes a seat, exactly as when it is created in the console. A plan that creates one beyond your seats, or on an edition without service accounts, reports an error.
Who can call the API:
| Action | Global role needed |
|---|---|
| Export, list managed objects | ADMIN, SECURITY_ADMIN or AUDITOR |
| Plan, apply | ADMIN or SECURITY_ADMIN |
| Release an object from management | ADMIN |
A SECURITY_ADMIN (Enterprise) cannot grant anything to itself, nor to a role group it belongs to.
See Roles and access control.
The YAML format
A declaration set is one of:
- several YAML documents separated by
---; - a JSON array of documents;
- a JSON object with an
itemsarray.
Every document has the same envelope:
apiVersion: kcm.kcmhub.io/v1
kind: RoleGroup # RoleGroup, ServiceAccount or RoleBinding
metadata:
name: payments-team # identifies the object within its kind
spec:
...
metadata.name accepts letters, digits and the characters . _ @ : -. A RoleGroup is
named after the role group, a ServiceAccount after the service account. A RoleBinding name is yours to choose: it
only identifies the binding in the files and in the plan.
RoleGroup
apiVersion: kcm.kcmhub.io/v1
kind: RoleGroup
metadata:
name: payments-team
spec:
description: Payments squad
enabled: true
members: [alice@example.com, bob@example.com]
| Field | Description |
|---|---|
description | Optional text. |
enabled | true by default. A disabled role group grants nothing. |
members | Usernames of existing users (local or SSO). Users are never created by the declarations. Memberships that come from your identity provider at SSO sign in are not listed here. |
ServiceAccount
apiVersion: kcm.kcmhub.io/v1
kind: ServiceAccount
metadata:
name: payments-ci
spec:
description: Deployment pipeline of the payments services
enabled: true
description is optional; enabled is true by default. Disabling a service
account stops its tokens from working and frees its seat.
RoleBinding
A role binding grants either a role on a resource, or a list of operations on resources matching a pattern.
# A role on a Kafka cluster
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
name: payments-team-editor-prod
spec:
subject: { kind: RoleGroup, name: payments-team }
role: editor # viewer, editor or admin
scope: /groups/prod/clusters/payments
---
# Operations on the topics starting with "payments."
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
name: payments-ci-topics
spec:
subject: { kind: ServiceAccount, name: payments-ci }
operations: [READ, WRITE, DESCRIBE]
patternType: PREFIXED # LITERAL (default) or PREFIXED
scope: /groups/prod/clusters/payments/topics/payments.
description: CI reads and writes the payments topics
| Field | Description |
|---|---|
subject | kind and name of who receives the grant, see
below. |
role | viewer, editor or admin, for a scope
that ends on a resource (KCM group, Kafka cluster, Kafka Connect cluster, ksqlDB endpoint). |
operations | For a scope that ends on a pattern (topics, consumer groups, subjects, connectors). Allowed values depend on the resource, see Scopes. |
patternType | LITERAL (default, exact name) or PREFIXED
(every name starting with the pattern). Pattern grants only. |
scope | Path of the resource, by names. |
enabled | true by default. |
description | Optional text. |
Subjects
subject.kind | subject.name |
|---|---|
User | An existing user, local or SSO. Users are not created by the declarations. |
ServiceAccount | A service account, existing or declared in the same set. |
RoleGroup | A role group, existing or declared in the same set. |
Scopes
A scope addresses a resource by its names, from the KCM group down. Grants add up: a role on a KCM group applies to its clusters, a role on a Kafka cluster gives every topic, consumer group, subject and connector of it, and pattern grants add resources to users who have no role there.
| Scope | Grants |
|---|---|
/groups/{group} | Role on a KCM group and everything below it. |
/groups/{group}/clusters/{cluster} | Role on a Kafka cluster, its Kafka Connect clusters and ksqlDB endpoints. |
.../clusters/{cluster}/connect/{connect} | Role on a Kafka Connect cluster. |
.../clusters/{cluster}/ksql/{endpoint} | Role on a ksqlDB endpoint (users and service accounts only, not role groups). |
.../clusters/{cluster}/topics/{pattern} | Topic operations: READ,
WRITE, CREATE, DELETE, ALTER, DESCRIBE,
ALL. |
.../clusters/{cluster}/consumer-groups/{pattern} | Consumer group operations:
DESCRIBE, DELETE, RESET_OFFSETS, ALL. |
.../clusters/{cluster}/schema-registries/{registry}/subjects/{pattern} | Subject
operations: DESCRIBE, CREATE, DELETE, ALTER,
ALL. |
.../clusters/{cluster}/connect/{connect}/connectors/{pattern} | Connector operations:
DESCRIBE, UPSERT, DELETE, CONTROL, ALL. |
The connector CONTROL operation covers pause, resume, restart and stop only. Updating or deleting a
connector needs UPSERT or DELETE.
Names with spaces, slashes or percent signs
Each name of the path (KCM group, cluster, Kafka Connect cluster, Schema Registry, ksqlDB endpoint) is one segment and is percent-decoded:
- a name containing
/or%must be written with%2For%25; - any other character may be percent-encoded too, for example
%20for a space. A space written as is also works, since the scope is a plain YAML string; - a
+stays a plus sign, it never means a space; - a malformed escape such as
%zzis refused with a clear error.
# KCM group "Data Platform", cluster "eu/west 1"
scope: /groups/Data%20Platform/clusters/eu%2Fwest%201
# the same group, space written as is
scope: /groups/Data Platform/clusters/eu%2Fwest%201
The pattern at the end of a topic, consumer group, subject or connector scope is taken
as is, without decoding: .../topics/orders/v1 is the topic pattern
orders/v1.
The export writes names with %2F and %25 only, so files exported from any 1.5.x
release keep their meaning.
Kafka Connect clusters, Schema Registries and ksqlDB endpoints must have unique names within their Kafka
cluster, so that a scope always points to one resource. At startup, KCM 1.5.0 renames the duplicates left by
older versions (name-2, name-3...), logs and audits each rename, and it refuses a
duplicate name when one is created or renamed.
Export, plan, apply
The usual cycle is:
- Export what KCM holds today, to start your repository from the real state.
- Edit the files and open a merge request.
- Plan: KCM compares the files with its state and lists the changes. Nothing changes.
- Apply after review: KCM makes the changes in one transaction.
The source (managedBy)
Plan and apply take a source parameter that names the declarations, for example a repository or a
pipeline: git:payments-access. It accepts letters, digits and . _ : / @ -. Every object
applied is marked managedBy: iac:{source}.
- An object managed by one source cannot be taken over by another source: the plan reports an error.
- Pruning only ever touches the objects of the same source.
- Use one source per repository (or per team folder) and keep it stable.
Export
GET /api/v1/iac/export?kinds=RoleGroup,ServiceAccount,RoleBinding&managedBy=iac:{source}&format=yaml
kinds: comma-separated list, all three by default.managedBy: only the objects of that source. Without it, every object is exported.format:yaml(default) orjson.
Managed objects keep their declared name. Objects not managed yet get a stable generated name, so exporting twice gives the same file.
curl -s -H "Authorization: Bearer $KCM_TOKEN" \
"https://kcm.example.com/api/v1/iac/export?format=yaml" > access.yaml
Plan
POST /api/v1/iac/plan?source={source}&prune=true|false
Send the declarations as the request body (Content-Type: application/yaml or
application/json). The answer lists each change and every error:
{
"managedBy": "iac:git:payments-access",
"applied": false,
"changes": [
{ "action": "CREATE", "kind": "RoleGroup", "name": "payments-team", "detail": "2 member(s)" },
{ "action": "ADOPT", "kind": "ServiceAccount", "name": "payments-ci", "detail": null },
{ "action": "NOOP", "kind": "RoleBinding", "name": "payments-ci-topics", "detail": null }
],
"errors": []
}
| Action | Meaning |
|---|---|
CREATE | The object does not exist and will be created. |
UPDATE | The object exists and differs from the declaration. |
DELETE | Only with prune=true: an object of this source is no longer
declared. |
ADOPT | An existing object matches the declaration: it becomes managed without any change. |
NOOP | Already managed and identical. |
Errors (unknown user, unknown cluster in a scope, operation not allowed for a resource, object owned by another
source, edition limits...) are listed in errors. A plan with errors cannot be applied.
Apply
POST /api/v1/iac/apply?source={source}&prune=true|false
Same body and same answer as the plan, with "applied": true. All changes are made in
one transaction: either every change is applied, or none. The call is refused with
400 when the plan has errors.
Prune
With prune=true, objects of this source that are no longer declared are removed:
- role bindings and role groups are deleted;
- service accounts are disabled, not deleted: their tokens stop working at once, and you can enable them again later.
Without prune (the default), removing a document from the files leaves the object in KCM, still
managed by the source. Run the plan with prune=true first to see what would go.
Managed objects are read only
Once applied, a managed role group, service account or role binding can only be changed through its declarations:
- the console shows it with an IaC label and does not let you edit or delete it;
- the REST API answers
409 Conflictwith error codeKCM-1006to any change of it.
A grant added in the console to a managed role group is not managed: the next plan with
prune leaves it alone, because only the objects of the source are pruned.
Drift
Any difference between the files and KCM shows in the next plan as an UPDATE, CREATE
or DELETE. To check drift on a schedule, run a plan and fail the job when it lists anything other than
NOOP.
Release an object
To hand an object back to the console, list the managed objects and remove the marking (global
ADMIN). The object is kept as is, becomes editable again and is no longer pruned by its source.
# Managed objects, optionally filtered by kind
curl -s -H "Authorization: Bearer $KCM_TOKEN" "https://kcm.example.com/api/v1/iac/managed?kind=RoleBinding"
# Release one object
curl -s -X DELETE -H "Authorization: Bearer $KCM_TOKEN" \
"https://kcm.example.com/api/v1/iac/managed/RoleBinding/payments-ci-topics"
Examples with curl
The examples use a token in $KCM_TOKEN whose account has the global ADMIN or
SECURITY_ADMIN role:
- from the Team edition on, use a service account
token (
kcm_sa_...) dedicated to the pipeline; - in Community and Starter, sign in as the administrator with
POST /api/auth/loginand use the returned token, see REST API.
KCM_URL=https://kcm.example.com
SOURCE=git:payments-access
# 1. Plan, without and with prune
curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
--data-binary @access.yaml "$KCM_URL/api/v1/iac/plan?source=$SOURCE&prune=true"
# 2. Apply
curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
--data-binary @access.yaml "$KCM_URL/api/v1/iac/apply?source=$SOURCE&prune=true"
# 3. Export what this source manages, to compare with the repository
curl -sf -H "Authorization: Bearer $KCM_TOKEN" \
"$KCM_URL/api/v1/iac/export?managedBy=iac:$SOURCE&format=yaml"
Use --data-binary rather than -d, so the line breaks of the YAML file are kept.
CI example: plan on merge request, apply on main
A typical repository holds one or more YAML files (here access/*.yaml). The pipeline plans on every
merge request and applies when the change reaches the main branch. Store the token as a masked, protected CI
variable.
GitLab CI
# .gitlab-ci.yml
variables:
KCM_URL: https://kcm.example.com
KCM_SOURCE: git:platform-access
# KCM_TOKEN: masked and protected CI/CD variable
.kcm:
image: alpine:3.20
before_script:
- apk add --no-cache curl jq
# one stream of YAML documents: a --- line between two files
- awk 'FNR==1 && NR!=1 {print "---"} {print}' access/*.yaml > all.yaml
kcm-plan:
extends: .kcm
stage: test
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
script:
- |
curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
--data-binary @all.yaml "$KCM_URL/api/v1/iac/plan?source=$KCM_SOURCE&prune=true" | tee plan.json
- jq -r '.changes[] | select(.action != "NOOP") | "\(.action)\t\(.kind)\t\(.name)\t\(.detail)"' plan.json
- test "$(jq '.errors | length' plan.json)" -eq 0 || (jq -r '.errors[]' plan.json; exit 1)
artifacts:
paths: [plan.json]
kcm-apply:
extends: .kcm
stage: deploy
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
resource_group: kcm-access
script:
- |
curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
--data-binary @all.yaml "$KCM_URL/api/v1/iac/apply?source=$KCM_SOURCE&prune=true" | tee apply.json
- jq -e '.applied' apply.json
GitHub Actions
# .github/workflows/kcm-access.yml
name: kcm-access
on:
pull_request:
paths: ["access/**"]
push:
branches: [main]
paths: ["access/**"]
env:
KCM_URL: https://kcm.example.com
KCM_SOURCE: git:platform-access
jobs:
kcm:
runs-on: ubuntu-latest
concurrency: kcm-access
steps:
- uses: actions/checkout@v4
- name: Merge the YAML files
run: awk 'FNR==1 && NR!=1 {print "---"} {print}' access/*.yaml > all.yaml
- name: Plan
if: github.event_name == 'pull_request'
env:
KCM_TOKEN: ${{ secrets.KCM_TOKEN }}
run: |
curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
--data-binary @all.yaml "$KCM_URL/api/v1/iac/plan?source=$KCM_SOURCE&prune=true" > plan.json
jq -r '.changes[] | select(.action != "NOOP") | "\(.action) \(.kind) \(.name)"' plan.json
test "$(jq '.errors | length' plan.json)" -eq 0 || { jq -r '.errors[]' plan.json; exit 1; }
- name: Apply
if: github.event_name == 'push'
env:
KCM_TOKEN: ${{ secrets.KCM_TOKEN }}
run: |
curl -sf -X POST -H "Authorization: Bearer $KCM_TOKEN" -H 'Content-Type: application/yaml' \
--data-binary @all.yaml "$KCM_URL/api/v1/iac/apply?source=$KCM_SOURCE&prune=true" | jq -e '.applied'
- Give the pipeline its own service account, with the global
SECURITY_ADMINrole on Enterprise, orADMINotherwise, and a token with a validity period. - Run one apply at a time (
resource_group,concurrency). - Start without
prune, check a few plans, then turn it on. - Do not declare the service account that runs the pipeline in the set it applies (create it in the console, or declare it under another source), so a mistake in the files cannot disable it.
Audit of the changes
Every call is recorded in the audit trail:
| Action | Recorded |
|---|---|
IAC_APPLY_REQUEST | Each apply call, with the source and whether
prune was set. |
IAC_APPLY | Each successful apply, with the source and the number of changes by kind of change. |
IAC_UNMANAGE | An object released from management, with its kind and name. |
The changes themselves (role groups, members, grants, service accounts) are also recorded as usual, attributed to the account that called the API. On Enterprise, these events can be exported to CSV and streamed to your SIEM.
Complete example
# access/payments.yaml
apiVersion: kcm.kcmhub.io/v1
kind: RoleGroup
metadata:
name: payments-team
spec:
description: Payments squad
members: [alice@example.com, bob@example.com]
---
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
name: payments-team-viewer-prod
spec:
subject: { kind: RoleGroup, name: payments-team }
role: viewer
scope: /groups/Production/clusters/payments
---
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
name: payments-team-editor-staging
spec:
subject: { kind: RoleGroup, name: payments-team }
role: editor
scope: /groups/Staging/clusters/payments
---
apiVersion: kcm.kcmhub.io/v1
kind: ServiceAccount
metadata:
name: payments-ci
spec:
description: Deployment pipeline of the payments services
---
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
name: payments-ci-topics
spec:
subject: { kind: ServiceAccount, name: payments-ci }
operations: [READ, WRITE, DESCRIBE, CREATE]
patternType: PREFIXED
scope: /groups/Production/clusters/payments/topics/payments.
---
apiVersion: kcm.kcmhub.io/v1
kind: RoleBinding
metadata:
name: payments-ci-groups
spec:
subject: { kind: ServiceAccount, name: payments-ci }
operations: [DESCRIBE, RESET_OFFSETS]
patternType: PREFIXED
scope: /groups/Production/clusters/payments/consumer-groups/payments-
Coming next
KCM as code will grow beyond access control: more resource kinds, and a Terraform provider built on the same
declarations. This section will describe them when they are released. Until then, only
RoleGroup, ServiceAccount and RoleBinding are accepted.