Skip to content

Creating and managing variants

A common task is to manage variants of the same configuration — for example, development, staging, and production variants, or per-region and per-tenant variants.

A variant is a space that was cloned from an upstream space — typically a base — and customized for a different environment, region, tenant, or experiment. Cloning links each unit in the variant space to the upstream unit it was copied from, so the variant can later be brought up to date as the upstream changes.

The cub variant commands manage variants at the space level and are the recommended way to work with them. Under the hood, the upstream relationships they establish are maintained per unit, and the lower-level unit commands remain fully supported — see working with variants at the unit level.

Managing variant spaces

The cub variant subcommands operate on a whole space and its units at once: seed a base space from rendered manifests, clone it into a variant, and later reconcile the variant with its upstream.

Seeding a base space with cub variant upload

When you already have rendered Kubernetes manifests — from the Helm plugin, kustomize build, helm template, or an external pipeline — cub variant upload ingests them into a space as units. It does not render anything; it stores what you give it.

kustomize build overlays/base | cub variant upload \
    --component web --variant base \
    --target web-base/cluster -

--component is required and names the component the space belongs to; --variant (default base) becomes the well-known Variant space label, and --stage, --environment, --region, --layer, and --owner set the rest. Set --stage for any space a ChangeWorkflow will promote into, since that is the label its Stages select on. The space slug comes from --space-pattern (default template:{{.Component.Slug}}-{{.Labels.Variant}}) or from an explicit --space, and the space is created if it does not exist. Every resource becomes its own unit, named for the resource rather than for the file it came from: a workload keeps its bare name (web), everything else takes its kind (web-service), and each AppConfig file becomes its own unit as well. The space records what it was uploaded from, in its ExternalSourceRef and ExternalSourceDigest annotations. --target binds the created units to a target. Rendered Secrets are never uploaded — apply them out of band.

--namespace is the release namespace: where namespaced resources that name no namespace, and cluster-scoped resources, belong. Give a component one namespace per space — that release namespace — and everything downstream stays unambiguous. A space already has a single delivery destination in its ReleaseTargetID, and set-namespace is space-wide, so cub variant create --namespace fills exactly one namespace in and needs no per-resource selection. --create-namespace synthesizes the Namespace resource when the bundle lacks it, and is off by default: a bare Namespace carries none of the pod-security labels, NetworkPolicy, ResourceQuota or LimitRange that make a namespace usable, and a platform usually provisions those together. Asking it to synthesize a Namespace named after a placeholder is an error, because that object must never reach a cluster. If the bundle already carries the release Namespace, none is synthesized and the upload reports the collision rather than failing.

A bundle that puts a stray resource in a second namespace — a RoleBinding in kube-system, say — is the case one namespace per space does not cover. Uploading it is fine, but cub variant create --namespace would move that resource too, so scope set-namespace yourself with --where-resource. It takes two invocations, because a metadata.namespace filter cannot match a cluster-scoped Namespace resource, and --where-resource has no OR:

# the namespaced resources in the release namespace
cub function do --space web-prod \
    --where-resource "metadata.namespace = 'confighubplaceholder'" \
    set-namespace web-prod
# and the Namespace resource itself
cub function do --space web-prod \
    --where-resource "ConfigHub.ResourceType = 'v1/Namespace' AND metadata.name = 'confighubplaceholder'" \
    set-namespace web-prod

This is the recommended way to seed a promotable base space.

Every upload is create-or-update, so there is no separate re-upload mode: units whose resources are unchanged are left alone, changed ones are merged, and new ones are created. A unit whose resource is no longer in the input is emptied rather than deleted — it keeps its id, its links and its history, and the next release withdraws the object — and an upload whose plan would empty anything asks first unless you pass --yes. The merge takes the difference between what the source uploaded last time and what it uploads now, and applies it to the unit as it is, much as kubectl apply merges with its last-applied configuration. A change you made in ConfigHub after the upload — a replica count, an annotation — survives it, unless the source changed the same field, in which case the source's new value wins. To hold a value even against that, protect the path: the upload then withholds the source's change and reports it as a conflict.

# Hold this value even when the source changes it
cub function set --space web-base --unit web --protect set-replicas 5

Letting the source win is usually what you want for a space whose content is generated elsewhere. If you find yourself protecting the same path after every render, the value belongs in the source instead.

Creating a variant with cub variant create

cub variant create clones an upstream space and all of its units into a new downstream space, linking each clone to its upstream unit so the variant can be promoted later.

cub variant create prod web-base \
    --environment Prod --region us-east2 \
    --target web-prod/cluster \
    --namespace web-prod \
    --unit-delete-gate critical --unit-destroy-gate critical

The first argument is the variant name, which becomes the new space's Variant label; the second is the upstream space to clone from. The new space's name defaults to <component>-<variant> from the cloned space's labels — web-prod here — and can be overridden with --space-pattern. The new space inherits the upstream's labels (with Variant overridden), along with its permissions. It selects the triggers the upstream selects: it inherits the upstream's WhereTrigger and TriggerFilterID, or, when the upstream has neither and so runs the triggers in it, gets a WhereTrigger selecting the upstream space's triggers. The triggers are not copied, so a trigger you add to the new space runs only if its selector matches it. The new space starts with no delete gates; --space-delete-gate adds gates, and then the upstream's gates are kept as well. --target retargets the cloned units and stamps the space's TargetID annotation, and --namespace replaces the placeholder namespace from the base so each variant lands in its own namespace. The UpstreamSpaceID annotation it records is what makes the space promotable.

To automatically customize the clones, define PostClone triggers in the upstream space, or select them with its WhereTrigger or TriggerFilterID. The new space selects the same triggers, so they run during the clone. See managing dependencies and validation and policies.

Promoting a variant with cub variant promote

cub variant promote reconciles a variant space with the upstream space recorded in its UpstreamSpaceID annotation. In one command it upgrades every unit whose upstream has advanced (merging the upstream changes), clones any units added to the upstream since the variant was created or last promoted, and copies the new units' links.

# Preview what would change first.
cub variant promote web-prod --dry-run -o mutations

# Promote, optionally within a changeset and with a change description.
cub variant promote web-prod \
    --changeset release-2024-06 \
    --change-desc "Promote web to prod"

A promotion takes the options a cub unit update does, and applies them to every write it makes, including the set-namespace that places units it adds:

  • --clearance names the guarded reasons the writes are cleared for. A guarded path it does not cover is withheld and reported as a conflict.
  • --guard KEY=VALUE records a reason on the paths written. --clearance has to cover it.
  • --subgroup records a category on the writes' mutations, so they can be found together afterwards.
  • --tag puts a tag on the revision each unit the promotion writes is left at, added units included. Units it does not change are not tagged. A variant with a unit to write that the tag already marks is not promoted, since a tag marks one revision of a unit.
  • --protect is refused, because a promotion is a merge, which is what protection holds paths against. See protecting a variant's own values.
cub variant promote web-prod --tag web-base/release-2024-06 --clearance owner=platform

Promotion only updates the configuration data; it does not make the change live. Publish a new Release afterward with cub release publish <space> so the GitOps operator picks it up.

A promotion that brings several upstream revisions records one revision in each unit for every upstream revision that has an effect there, re-running the functions each one recorded (see merging and rebasing). So there is no single revision before the promotion to go back to, and cub unit update --restore -1 undoes only its last step. Give the promotion a name to undo it by: promote within a ChangeSet with --changeset, and restore to Before:ChangeSet:<slug>, or promote a ChangeOrder with --change-order, and take it back out with cub variant demote. --squash merges each unit's range as one revision instead.

To promote a named change rather than everything the upstream has reached — one image bump, through dev then staging then prod, gated on each stage actually running it — pass --change-order. See tracking and promoting changes.

Only spaces created by cub variant create can be promoted this way. For finer-grained or cross-space promotions, use the unit-level --upgrade mechanism described under keeping variants in sync, typically wrapped in a ChangeSet.

Protecting a variant's own values

A variant exists to differ from its base. The question every promotion asks is which of those differences are the variant's to keep and which are values it is only carrying until the upstream changes them — and ConfigHub answers it per path, from what you have said, rather than by guessing from the data.

Protection is opt-in. An ordinary change — a hand edit, a function invocation, a trigger, a needs/provides binding — leaves each path it writes as protected as it found it, and content that arrives from a clone, an upgrade, or a merge is recorded unprotected. So by default an upstream change to a path reaches the variant even if someone here had set that path to something else. That is the right default for a value the variant is carrying: a replica count copied down from the base should follow the base when the base changes its mind.

A value the variant has chosen, on the other hand, should say so. Two ways to say it:

As you make the change, with --protect on cub unit update, cub function set, or cub run. Every path the change writes becomes a protected local override. A merge is refused --protect — an upgrade, a --merge-source from another unit, a --merge-external-source, a --resolve of an UpgradeUnit or MergeUnits link, and a promotion — since it is what protection holds paths against: claiming what it delivered would stop the upstream from updating it, and a merge that replays several upstream revisions would withhold its own second one. A --merge-source Self revert may protect the values it puts back.

cub function set --space prod --unit backend-clone --protect set-replicas 5

Afterwards, per path, with cub unit set-protection. Each --protect and --unprotect names a resource type, a resource name, and a path:

# This deployment's replica count is prod's own from here on.
cub unit set-protection --space prod backend-clone \
    --protect "apps/v1/Deployment:prod/backend:spec.replicas"

# Changed our mind: let the base drive it again.
cub unit set-protection --space prod backend-clone \
    --unprotect "apps/v1/Deployment:prod/backend:spec.replicas"

--unprotect re-opens a path, so the next merge overwrites it with whatever the upstream has. Either form creates a new revision only if a flag actually changes, and neither touches the configuration data — protection is a statement about who owns a field, not a value.

Leave --protect off for a script propagating a release by any other means, such as invoking functions across variants: protecting what it wrote would turn upstream content into local overrides and block every merge after it.

To see what a unit currently protects, ask for its mutations. The two lists are exactly this distinction — what the variant owns, and what a merge is free to update:

cub unit get --space prod backend-clone -o mutations

Locally overridden (preserved during merges):
Resource: apps/v1/Deployment prod/backend
  ~ [Update] spec.replicas  (#2)

Eligible for upstream merges:
Resource: apps/v1/Deployment prod/backend
  + [Add] (#1)

Protection is stored per path, so it survives a restore and is remembered across merges. It belongs to the resource rather than to one API version of its type: when an upgrade moves a resource to a new apiVersion, such as a SecretStore from external-secrets.io/v1beta1 to external-secrets.io/v1, the paths the variant protected stay protected, and a resource type in --protect or --unprotect names the resource in any of its versions. Two mechanisms decide the same question by rule instead — a WhereMutation filter over mutation history, which protects further paths on top of the stored flags, and the subtraction step, which infers overrides from the diff at merge time and replaces the stored flags for the merges it applies to.

Protection applies only to merges from another unit. It does not stop a hand edit, a function invocation, or a trigger in the variant itself from changing the path, and a merge of the unit's own history (--merge-source Self) ignores it. To constrain those as well, use a guard.

Guarding values against every change

A guard records why a path holds the value it does, as a KEY=VALUE pair, and every change to the unit's configuration data — a hand edit, a function invocation, a trigger, a link, an upgrade or merge, cub k8s refresh — must be cleared for that reason before it may overwrite the path. A change states what it is cleared for with a clearance. A change without a matching clearance leaves the guarded path as it was, applies the rest, and reports what it withheld as a conflict.

The mechanism is modeled on Kubernetes taints and tolerations: a guard on a path plays the part of a taint on a node, and a clearance on a change plays the part of a toleration on a pod. The difference is that a guarded value is not undesirable — it was set deliberately, and the guard is there so that a tool or person who does not know why does not change it by accident.

Protection Guards
Constrains merges from another unit every change to the unit's data
Records that the path is a local override the reasons the path holds its value
A change gets past it by nothing; unprotect the path, or apply the conflict a clearance that covers the path's guards
Covers descendants the nearest protected or unprotected ancestor decides every guard on every ancestor applies
Reaches variants no; each variant protects its own values yes; clones copy guards, and merges carry upstream guard changes

Use protection for a value a variant chose for itself and wants to keep out of promotions. Use a guard when the value has a reason that anyone changing it should know about, wherever the change comes from.

Choosing guard keys and values

The key names a class of reason and is what a clearance matches on; the value distinguishes cases within that class. Two shapes cover most uses:

  • owner=<what maintains it> — the field is maintained by something specific, such as a link or a trigger, and other writers should leave it alone. owner=transform-link on an image field that a TransformPaths link sets is the typical case.
  • policy-exception=<which policy> — the value deliberately violates a default policy, and a tool that normalizes values to that policy must not undo it. policy-exception=host-network on a pod spec that needs host networking, or policy-exception=run-as-root.

Keep the set of keys small and shared across your organization: a clearance is usually written by someone who has not looked at the unit, so it can only name reasons that mean the same thing everywhere. Keys and values are letters, digits, - and _ (values may also contain .), up to 128 characters. Keys starting with confighub- are reserved. A guard value is matched, not read, so keep explanations in the change description rather than in the guard.

Setting guards

As you make the change, with --guard KEY=VALUE on cub unit update, cub function do, cub run, cub k8s refresh, or cub variant promote. The guard is recorded on every path the change wrote, just as --protect marks them. The change has to pass a --clearance that covers its own guards, or it is refused: a change is withheld by every guard it is not cleared for, its own included, so the next change that writes the path the same way would be withheld, and so would the second upstream revision of an upgrade that replays several.

cub function do --space prod --unit backend \
    --guard policy-exception=capacity --clearance policy-exception=capacity set-replicas 7

Triggers and links take --guard too, and record it on the paths they write each time they run. A TransformPaths link that maintains a field is the natural place to say why. Give it the matching clearance as well, as for any change; a trigger or link whose guards its clearance does not cover is refused when it is created or updated:

cub link create --space prod image-link backend images \
    --update-type TransformPaths --from-stdin < image-link.yaml \
    --guard owner=transform-link --clearance owner=transform-link

UpgradeUnit and MergeUnits links refuse --guard, because their guards arrive from the upstream unit.

Afterwards, or ahead of time, per path, with cub unit set-guard. Each --guard names a resource type, a resource name, a path, and the guard; leave the path empty to guard the whole resource:

# A link maintains this image
cub unit set-guard --space prod backend \
    --guard "apps/v1/Deployment:prod/backend:spec.template.spec.containers.?name=app.image=owner=transform-link"

# This pod spec needs host networking on purpose
cub unit set-guard --space prod backend \
    --guard "apps/v1/Deployment:prod/backend:spec.template.spec=policy-exception=host-network"

A guard applies to the path and everything under it, including paths that do not exist yet: guarding spec.template.spec covers a container added later. For the same reason, set-guard accepts a path, or a resource, that the unit does not currently have, so check the spelling — a misspelled path is accepted and guards nothing.

Setting guards changes no configuration data. It creates a new revision if the guards change, and neither --guard on a change nor set-guard --guard removes a guard. To remove one, use --remove-guard, which takes the same form without the value, together with a clearance for the guard being removed:

cub unit set-guard --space prod backend \
    --remove-guard "apps/v1/Deployment:prod/backend:spec.template.spec=policy-exception" \
    --clearance policy-exception=host-network

Changing or removing the guards at a path that already has some needs a clearance for them; setting the first guard on a path does not.

Clearances

A change that needs to write a guarded path says which reasons it is cleared for with --clearance, on cub unit update, cub function do, cub run, cub k8s refresh, and cub unit set-guard, and on triggers and links, which run without anyone passing a flag. The syntax follows Kubernetes set-based label selectors:

Clearance Clears a guard KEY=VALUE when
KEY always, whatever the value
KEY=A;B the value is A or B
KEY!=A;B the value is neither A nor B
!KEY never: a path carrying any KEY guard is not written, whatever else the clearance says

--clearance is repeatable, and a guarded path is written only if every guard key on it and on its ancestors is cleared. A change with no clearance writes no guarded path. Paths with no guards are unaffected, and a unit with no guards at all behaves exactly as if guards did not exist.

# Withheld: set-replicas is not cleared for the capacity exception
cub function do --space prod --unit backend set-replicas 42

# Lands
cub function do --space prod --unit backend \
    --clearance policy-exception=capacity set-replicas 42

# Cleared for any owner, but never touch a policy exception
cub function do --space prod --where "Labels.tier = 'backend'" \
    --clearance owner --clearance '!policy-exception' set-image app registry.example.com/app:1.4

Give a trigger or a link the clearance for the guards it is meant to overwrite. Guard an image field with owner=transform-link, give the TransformPaths link that maintains it --clearance owner=transform-link, and give the UpgradeUnit link none: upgrades then stop overwriting the image, and the transform link keeps updating it.

cub link update --space prod --patch image-link --clearance owner=transform-link

A clearance says the change knows about a reason; it does not grant permission. Anyone who can edit the unit can pass one. Guards exist to stop changes made without knowing why a value is what it is, and the clearance a function invocation ran with is recorded on its mutation, so overriding a guard is visible in the unit's history:

cub mutation list --space prod backend --where "FunctionInvocation.Clearance.*.Key = 'policy-exception'"

Restricting which users may clear which keys may be added in the future.

Guards on variants

A guard on a base is policy about that configuration, so variants inherit it. cub variant create and unit clones copy guards along with the data, and an upgrade, a promotion, or a MergeUnits link carries the guards added, changed, or removed upstream since the last merge — including a guard change with no data change beside it. When the base drops a policy exception because the workload no longer needs it, the variants drop it too.

Guard changes arrive under the same rule as data: an upstream guard change lands at a path only if the merge is cleared for the guards the variant already has there. A guard change the merge was not cleared for is reported as a GuardWithheld conflict, so a variant that did not receive a base's new policy exception shows it, rather than differing silently. A variant can still add or remove guards of its own with set-guard.

Seeing guards

cub unit blame marks guarded fields G in its FLAGS column, next to protection's P, and --verbose lists the guards on each field:

cub unit blame --space prod backend --verbose

cub unit get --verbose prints the unit's whole guard table under Path Annotations. Triggers and links that set guards, or carry clearances, are selectable:

cub trigger list --where "Guards.owner = 'platform'"
cub link list --where "Clearance.*.Key = 'owner'"

Conflicts: what a merge could not apply

A merge that cannot apply part of what it brought does not fail, and does not apply it anyway. It applies the rest and records what it withheld on the unit, where it stays until you deal with it. On a unit with guards, any change can withhold part of what it wrote in the same way:

cub unit conflicts --space prod backend-clone

Each conflict names the resource, the path, the value that was withheld, and a reason:

  • ProtectedPath — the path is a protected local override, so the upstream's change to it was withheld. This is the common one, and it is the protection mechanism reporting itself rather than working silently.
  • Guarded — the path carries a guard the change was not cleared for, so the change's value was withheld. The conflict names the guard.
  • GuardWithheld — the upstream added, changed, or removed a guard at a path, and the merge was not cleared for the guards this unit already has there, so the guard change was withheld. Applying it writes the guard change rather than any configuration data.
  • ExclusiveWithheld / ExclusiveCleared — the two sides set mutually exclusive fields (a volume with one source, a deployment strategy and its options). The downstream's choice stands and the upstream's is reported.
  • Subtracted, DeleteShadowed — the subtraction step dropped a change, or a deletion could not be applied because the target had changed underneath it.
  • UnresolvedPath — the path the source changed could not be located in this unit.
  • ReplayFailed — a replayed function errored here. The step was patched instead, and the conflict carries the function name and the error.

Conflicts are queryable, so "which variants have something outstanding?" is one command:

cub unit list --where "Conflicts.*.Reason = 'ProtectedPath'"

Deal with one by applying it — take the upstream's value after all — or dismissing it:

# What would taking the upstream's value do here? --dry-run writes nothing.
cub unit conflicts --space prod backend-clone --apply --dry-run -o mutations

# Apply everything withheld for a given reason
cub unit conflicts --space prod backend-clone --apply --reason ProtectedPath

# Or drop the report without changing any data
cub unit conflicts --space prod backend-clone --dismiss --reason ProtectedPath

--path and --resource narrow the selection further; with no selector, --apply and --dismiss act on everything outstanding. --dry-run works with either and reports what the request would do — with -o mutations, as the same per-path list a real one records.

Applying writes the withheld value and records the path as content that came from elsewhere, so a later upstream change to it lands normally rather than reporting the same conflict release after release. Dismissing changes no data and leaves the protection or guard in place. Applying a Guarded conflict needs no clearance — applying it is the decision the clearance exists to ask for — and leaves the guard in place, since a clearance acknowledges a reason rather than retiring it.

A merge replaces the outstanding set rather than adding to it, so a later merge that lands cleanly clears what an earlier one withheld — the set describes where the unit stands now, not everything that ever happened. Within a single walked range the conflicts accumulate across the hops, so nothing is erased before you have seen it.

To make outstanding conflicts block a Release rather than sit there, attach the vet-no-merge-conflicts validating function as a Trigger; it fails while anything is outstanding, which turns into a Validation Error.

Base units

You may choose to maintain base units which you copy to create all of the variants you plan to deploy. Base units are similar to abstract classes in that they aren't intended to be deployed. For example, they need not have attached deployment targets. A unit without a target is never in a Release, and you can select only units with targets with TargetID IS NOT NULL, or skip base units with labels.

This approach enables a clearer separation of common attributes and custom attributes.

Placeholders for undetermined values

Base units, and sometimes other new config units, contain default values for most fields, but some may require specific values that cannot yet be provided. This is indicated by placeholders.

A placeholder is a special string or number in the config data which indicates that you must replace it with a real value before the configuration is valid. ConfigHub uses the string "confighubplaceholder" to indicate where a string field needs to be supplied with a value and 9 9s (999999999) for integer fields.

The vet-placeholders function can be installed as a trigger to prevent applying configuration units with unreplaced placeholders, and get-placeholders can return a list of field paths containing placeholders.

Triggers are good for setting general properties of an environment: scale/cost (e.g., replicated vs not, backups vs not), privacy (e.g., public vs not, data with real PII vs not), security (e.g. must be encrypted vs not, can't run as root), and the like.

Links better address more specific resource properties like hostnames, IP addresses, etc.

See managing dependencies for guidance regarding how to replace placeholders with variant-specific values.

Going further