Skip to content

Creating and managing variants

A common task is to manage variants of the same configuration — for example, development, staging, and production variants, or per-region and per-tenant variants.

A variant is a space that was cloned from an upstream space — typically a base — and customized for a different environment, region, tenant, or experiment. Cloning links each unit in the variant space to the upstream unit it was copied from, so the variant can later be brought up to date as the upstream changes.

The cub variant commands manage variants at the space level and are the recommended way to work with them. Under the hood, the upstream relationships they establish are maintained per unit, and the lower-level unit commands described later in this guide remain fully supported.

Managing variant spaces

The cub variant subcommands operate on a whole space and its units at once: seed a base space from rendered manifests, clone it into a variant, and later reconcile the variant with its upstream.

Seeding a base space with cub variant upload

When you already have rendered Kubernetes manifests — from the installer, kustomize build, helm template, or an external pipeline — cub variant upload ingests them into a space as units. It does not render anything; it stores what you give it.

kustomize build overlays/base | cub variant upload \
    --component web --variant base \
    --granularity per-resource \
    --target web-base/cluster -

--component and --variant are required and become the well-known Component and Variant space labels; --environment, --region, --layer, and --owner set the rest. The space slug comes from --space-pattern (default template:{{.Labels.Component}}-{{.Labels.Variant}}) or from an explicit --space, and the space is created if it does not exist. --granularity per-resource creates one unit per resource; the default minimal packs everything into a single unit, splitting out CRDs and each AppConfig file. --target binds the created units to a target. Rendered Secrets are never uploaded — apply them out of band.

This is the recommended way to seed a promotable base space; for other rendered-manifest workflows, see importing rendered manifests.

Creating a variant with cub variant create

cub variant create clones an upstream space and all of its units into a new downstream space, linking each clone to its upstream unit so the variant can be promoted later.

cub variant create prod web-base \
    --space-pattern "template:{{.Labels.Component}}-{{.Labels.Variant}}" \
    --environment Prod --region us-east2 \
    --target web-prod/cluster \
    --namespace web-prod \
    --unit-delete-gate critical --unit-destroy-gate critical

The first argument is the variant name, which becomes the new space's Variant label; the second is the upstream space to clone from. The new space inherits the upstream's labels (with Variant overridden), along with its WhereTrigger, TriggerFilterID, permissions, and delete gates. --target retargets the cloned units and stamps the space's TargetID annotation, and --namespace replaces the placeholder namespace from the base so each variant lands in its own namespace. The UpstreamSpaceID annotation it records is what makes the space promotable.

To automatically customize the clones, define PostClone triggers and select them via the upstream space's WhereTrigger or TriggerFilterID so they are copied down and run during the clone. See managing dependencies and validation and policies.

Promoting a variant with cub variant promote

cub variant promote reconciles a variant space with the upstream space recorded in its UpstreamSpaceID annotation. In one command it upgrades every unit whose upstream has advanced (merging the upstream changes), clones any units added to the upstream since the variant was created or last promoted, and copies the new units' links.

# Preview what would change first.
cub variant promote web-prod --dry-run -o mutations

# Promote, optionally within a changeset and with a change description.
cub variant promote web-prod \
    --changeset release-2024-06 \
    --change-desc "Promote web to prod"

Promotion only updates the configuration data; it does not apply the change to live resources. Apply it afterward with cub unit apply, or — when a GitOps operator pulls from the OCI endpoint — publish a new release with cub release publish.

Only spaces created by cub variant create can be promoted this way. For finer-grained or cross-space promotions, use the unit-level --upgrade mechanism described under keeping variants in sync, typically wrapped in a ChangeSet.

Working with variants at the unit level

The cub variant commands above build on per-unit upstream relationships. You can also work with those relationships directly, which is useful for partial scopes, cross-space fleets, and reshaping upstream links after the fact. ConfigHub offers several merge-based mechanisms for keeping related units in sync. Pick the one that matches your goal:

Goal Mechanism
Inherit config properties Clone/upgrade, UpgradeUnit Links
Repeatedly render or generate config data MergeUnits Links
Combine or split config units, with updates MergeUnits Links
Repeatedly upload config data --merge-external-source
Merge a ChangeSet into other variants or upstream --merge-source

Cloning units individually

In ConfigHub you can copy (known as clone in the UI) config units in a way that tracks their relationship in order to help you keep them in sync. The new copied unit is said to be downstream from the original upstream unit.

Different deployment environments would have different spaces corresponding to them, so typically one would clone a unit to one or more other spaces.

You can copy one unit to another space with the CLI like this:

cub unit create --space prod --upstream-space dev --upstream-unit backend

You could copy multiple units to multiple spaces also. To copy all units from dev into multiple prod spaces (e.g., corresponding to multiple clusters or regions), you could do something like:

cub unit create --space dev --where-space "Labels.Environment = 'prod'"

When a unit is cloned, ConfigHub records the upstream unit and the upstream revision the clone was created from on the new downstream unit (UpstreamUnitID and UpstreamRevisionNum), and automatically creates an UpgradeUnit Link from the clone (downstream) to the original (upstream). The Link tracks which upstream revision was last merged and carries a WhereMutation filter that constrains which downstream changes can be overwritten by future upgrades. The default WhereMutation only allows mutations that originated from the clone or a prior upgrade or merge — anything else (like a hand edit, function invocation, or trigger output) is treated as a local override and preserved.

Establishing or changing the upstream relationship after creation

A unit's UpstreamUnitID is read-only and cannot be set or modified directly with cub unit update. To establish, change, or remove the upstream relationship after a unit has been created, manipulate the unit's UpgradeUnit Link.

To establish a relationship between two units that already exist, create an UpgradeUnit Link with --make-current. --make-current pins the Link's UpstreamLastMergedRevisionNum and DownstreamLastMergedRevisionNum to the current head revisions of the two units, so the next upgrade only merges changes made after this point — it does not retroactively merge the upstream's full history into the downstream:

cub link create --space prod --update-type UpgradeUnit --make-current upgrade-mydown mydown myupstream

To change which unit is upstream, delete the existing UpgradeUnit Link and create a new one. Deleting an UpgradeUnit Link clears the downstream unit's UpstreamUnitID and UpstreamRevisionNum. Without --make-current, creating a new UpgradeUnit Link performs an initial merge using the new upstream's full history.

Note: a unit can have at most one outgoing UpgradeUnit Link.

Reversing the upstream relationship

You may want to invert which unit is upstream. A common case: you started with a running app, then decided to copy it to a base unit so you can derive other variants from the base. The original (now an arbitrary variant) should become downstream of the base.

Reverse an UpgradeUnit Link with cub link update --reverse:

cub link update --space prod --patch --reverse upgrade-mydown

When the downstream and upstream units are in the same space, the Link is reversed in place. For cross-space Links, ConfigHub creates new reversed Link copies in the other space and deletes the originals (because a Link must live in the same space as its FromUnit).

Reversing also clears the Link's WhereMutation, since the heuristic that scoped it to changes "from the original upstream" no longer applies.

For bulk reversal — for example, after copying every unit in a dev space to a new base space and wanting the dev units to become downstream of the base — use cub link create --reverse against the original UpgradeUnit Links, then delete the originals:

# Copy every unit from app-dev to app-base; this auto-creates UpgradeUnit
# links in app-base pointing back to app-dev.
cub unit create --space app-dev --where "Slug LIKE '%'" --dest-space app-base

# Reverse: create reversed UpgradeUnit links in app-dev pointing to app-base.
cub link create --space app-base --where "UpdateType = 'UpgradeUnit'" --reverse

# Delete the now-redundant original links in app-base.
cub link delete --space app-base --where "UpdateType = 'UpgradeUnit'"

Keeping variants in sync

The upstream and downstream units can all be modified independently, but if you want to propagate a change from the upstream unit to the downstream units, ConfigHub uses the UpgradeUnit Link's tracked revision to know which upstream changes still need to be merged.

To upgrade and copy more recent changes from the upstream unit, you can execute a command like this:

cub unit update --space "*" --patch --upgrade --where "UpstreamUnit.Slug = 'backend' AND Space.Labels.Environment = 'prod'"

Alternatively, since the relationship is represented as a Link, you can also upgrade via the resolve mechanism:

cub unit update --space prod --patch --resolve "Link:*" backend-clone

Both approaches produce the same result. The --upgrade flag uses the UpgradeUnit Link's WhereMutation filter if one is set. The --resolve approach works with any Link of type UpgradeUnit or MergeUnits.

Upgrading the units modifies the configuration data but does not automatically apply the changes. To preview the changes first, add --dry-run. To inspect the resulting diff, add -o mutations:

cub unit update --space prod --patch --upgrade --dry-run -o mutations backend-clone

Merging upstream changes preserves changes made independently in the downstream unit — they are treated as overrides. ConfigHub supports this by tracking the source of each granular change via Mutation metadata stored on the unit (MutationSources), described in How merging works below.

Ad hoc merges across variants and from downstream to upstream

Upgrade propagates changes from the upstream unit to selected downstream units. For other directions — between sibling variants, from a downstream back to its upstream, or from a previous range of revisions of the same unit (for rebase-like flows) — use --merge-source with --merge-base and --merge-end. The base/end pair specifies a revision range on the source unit to merge into the target unit:

cub unit update \
    --patch \
    --space prod-west \
    --merge-source prod-east/backend \
    --merge-base Before:ChangeSet:prod-east/prodfix42 \
    --merge-end ChangeSet:prod-east/prodfix42 \
    backend

To apply a range of changes from a unit to itself (for rebase-like flows after a restore), use --merge-source Self:

cub unit update \
    --patch \
    --space acme-dev \
    --filter acme-home/acme-app \
    --merge-source Self \
    --merge-base Before:ChangeSet:acme-home/release-v452 \
    --merge-end ChangeSet:acme-home/release-v452 \
    --change-desc "Reapply release 452"

See merging and rebasing for additional examples.

Merging external sources into ConfigHub

When the base for a unit lives outside ConfigHub — for example, a YAML file in a Git repository, a chart from a registry, or a manifest produced by an external pipeline — use --merge-external-source with cub unit create and cub unit update to bring in changes from that source while preserving anything ConfigHub-side functions, links, or hand edits added on top:

# Create the unit, recording deployment.yaml as the external source.
cub unit create --space prod mydep deployment.yaml --merge-external-source deployment.yaml

# Apply ConfigHub-side defaults and customizations.
cub function do --space prod --unit mydep set-pod-container-security-context-defaults

# Later, after the upstream file changes, merge the new version in.
cub unit update --space prod mydep deployment-v2.yaml --merge-external-source deployment.yaml

The first invocation records deployment.yaml as the external source so subsequent updates with the same --merge-external-source are treated as continued merges from the same logical source rather than full replacements. Mutations made on the ConfigHub side (the security-context function above, link resolutions, etc.) are preserved through the merge.

By default the merge uses the latest revision whose source is MergeExternal as its base — i.e., the last externally-merged version of the file. Pass --merge-base to override that selection when you need to rebase the merge against a different revision of the unit (for example, to re-apply changes from an older external snapshot, or to reset the cursor after rewriting external history). It accepts the same revision syntax as --restore and --merge-base for --merge-source:

cub unit update --space prod mydep deployment-v2.yaml \
    --merge-external-source deployment.yaml \
    --merge-base Tag:initial-external

--merge-end is not used with --merge-external-source; the new external data passed on the command line plays that role.

How merging works

--upgrade, --resolve against an UpgradeUnit/MergeUnits Link, --merge-source, and --merge-external-source all share the same merge engine. Internally, it is a four-way merge:

  1. Diff the source between two source revisions — for upgrade and MergeUnits, that range is the Link's UpstreamLastMergedRevisionNum to the upstream unit's current HeadRevisionNum; for --merge-source it's --merge-base to --merge-end; for --merge-external-source it's the previously-recorded external file (or the revision selected by --merge-base) to the new external file.
  2. Diff the downstream unit between the previously-merged downstream revision and its current head, so that the downstream's existing local differences from the source baseline are recognized.
  3. Subtract any changes in (1) that conflict with the downstream's local differences from (2) — those are the local "overrides" that get preserved by default.
  4. Apply the remaining changes as a patch to the downstream unit's head.

Step (3) is on by default for cross-unit merges (--upgrade, --resolve against an UpgradeUnit/MergeUnits Link, and a cross-unit --merge-source). It is always skipped for a self merge (--merge-source Self, where the source and target are the same unit), because subtracting the very changes you are reapplying would defeat the operation. You can also turn it off explicitly — see Disabling the subtraction step, which makes the stored Predicate values the sole way overrides are preserved.

UpstreamRevisionNum on the unit and UpstreamLastMergedRevisionNum on the Link are updated when the merge completes. UpstreamLastMergedRevisionNum can be edited if you need to advance or rewind the merge cursor without doing a real merge.

MutationSources: which mutation last touched each path

Every time a unit's data changes, ConfigHub records which mutation last set each configuration path on the unit's MutationSources. That includes hand edits, function invocations (whether ad hoc, triggered, or part of a clone or upgrade), needs/provides resolutions, and merge results. MutationSources is what lets the merge engine tell "the upstream changed this field" apart from "the downstream changed this field locally."

A few consequences worth knowing:

  • Restore preserves provenance. cub unit update --restore rewinds both the unit's Data and its MutationSources to the chosen revision, so subsequent merges treat the restored state as the truth about who-touched-what.
  • You can "reset" MutationSources by removing all data and adding it back. Updating a unit to empty config and then back to the desired config replaces every entry in MutationSources with the new mutation. This is the supported way to drop accumulated provenance when you want a future upgrade to overwrite paths it would otherwise consider local overrides.
  • Closing a difference re-opens the path. If you change a field on the downstream so it again matches the upstream, the field is no longer a difference between them, so a subsequent upstream change to that field will propagate (assuming WhereMutation allows it). There is no "sticky" override list, except through WhereMutation, described below — overrides are inferred from the diff at merge time.
  • Setting an override to the same value the upstream already has is not an override. It looks identical to the merged-in upstream value, so the merge engine has nothing to subtract.

Each path in MutationSources also carries a Predicate flag that marks whether it is eligible to be overwritten by a merge: true for values that came from a clone, upgrade, or merge, and false for local overrides (hand edits, functions, triggers, needs/provides). ConfigHub maintains these flags automatically. They are consulted when a merge runs without a WhereMutation and with the subtraction step disabled (below), and you can set them directly to change what a future merge may overwrite:

# Protect the downstream replica count from future merges.
cub unit set-predicates --space prod backend-clone \
    --predicate "apps/v1/Deployment:prod/backend:spec.replicas=false"

Setting a predicate to false protects that path; setting it back to true re-opens it. Changing predicates creates a new revision only if a flag actually changes. See the Mutation Sources concept for the full picture.

WhereMutation: scoping which downstream paths can be overwritten

The merge engine narrows step (3) using the Link's WhereMutation (or the request's --where-mutation). It is a filter over the downstream unit's mutation history that selects which downstream paths are eligible to be overwritten by the source-side changes.

The auto-created UpgradeUnit Link sets a default WhereMutation that allows only mutations whose source was a clone, upgrade, or merge from this same upstream. The practical effect: a downstream change made by cub function do, a hand edit, a needs/provides binding, or a trigger is a local override and is preserved through --upgrade. Mutations from the original clone or prior upgrades are upgrade-eligible and will be overwritten by newer upstream changes.

WhereMutation is empty by default in two cases:

  • The Link was created with cub link create (rather than auto-created during a clone).
  • The Link was reversed via cub link update --reverse or cub link create --reverse.

An empty WhereMutation means the whole downstream unit is eligible to be overwritten — every conflict is resolved in favor of the source. To get the upgrade-style "preserve local overrides" behavior on these Links, you have to populate WhereMutation yourself. To go the other direction and clear it on a Link that has one set:

cub link update --space prod --patch --where-mutation="" upgrade-backend

WhereMutation can also be used proactively to "protect" parts of a unit from upgrade — for example, a Link whose WhereMutation excludes mutations under spec.replicas will never overwrite the downstream's replica count.

Disabling the subtraction step

The subtraction in step (3) of How merging works is the default mechanism for preserving local overrides, but you can turn it off:

  • Per request, with --merge-disable-subtraction on cub unit update / cub unit update --patch.
  • Per Link, with cub link create/cub link update --merge-disable-subtraction (the Link's MergeDisableSubtraction field), which applies whenever that Link is resolved (including via --upgrade).

When subtraction is disabled, overrides are preserved instead by the stored Predicate values on MutationSources — paths whose Predicate is false are treated as protected local overrides and are not overwritten. (If a WhereMutation is also supplied, it still applies and takes the place of the stored predicates.)

Two cases worth knowing:

  • A self merge (--merge-source Self) always runs with subtraction disabled, regardless of the flag, since it is reapplying a range of the unit's own changes.
  • A cross-unit --merge-source subtracts by default, like --upgrade. Disable it when the merge base is not a true common ancestor of the source and target — for example, merging between independent units — where you want a straight patch of the source's changes rather than a three-way merge:
cub unit update \
    --patch \
    --space prod-west \
    --merge-source prod-east/backend \
    --merge-base Before:ChangeSet:prod-east/prodfix42 \
    --merge-end ChangeSet:prod-east/prodfix42 \
    --merge-disable-subtraction \
    backend

WhereResource: scoping which upstream resources participate

WhereResource on a Link (or --where-resource on the operation) filters the source side: only resources matching the expression contribute to the merge. It is useful for splitting a single upstream unit across multiple downstream units (e.g., separating CustomResourceDefinitions from the rest), or for narrowing an upgrade to a subset of resources.

Strategic merge patch and merge keys

ConfigHub treats configuration as data, not as text, so it merges resource-by-resource and field-by-field — independent edits to different fields of the same resource don't conflict. For Kubernetes resources, ConfigHub uses its own implementation of strategic merge patch, which is aware of Kubernetes merge keys (the x-kubernetes-patch-merge-key markers on associative lists like containers, volumes, env, and ports).

This has a few important implications:

  • Multi-line string fields are merged as text, except for embedded JSON. Free-form text (like a shell script in a ConfigMap value) is merged line-by-line rather than character-by-character. ConfigHub auto-detects JSON-valued strings and merges them as structured data instead. For configuration files that you want managed as structured data — properties, env, INI, TOML, YAML, JSON — store them in their own units rather than embedding them as strings in a ConfigMap; see application configuration for the recommended pattern.
  • Renames look like delete + add. Changing the name of a resource, or the merge key of a list element (e.g., a container's name), looks to the merge engine like a deletion of the old element followed by addition of a new one. ConfigHub attempts to detect renames via a similarity heuristic — if the bodies match closely, it carries the renamed element forward in place and preserves any independent downstream overrides on it. If they differ enough to fall outside the heuristic, the downstream sees a delete + add and any downstream overrides on the deleted element are lost.
  • Reordering may not do what you expect. ConfigHub tries to preserve the order of merge-keyed list elements, but reordering elements both upstream and downstream can produce surprising merge results. If you care about ordering, change it on one side at a time.

If you're unsure how a merge will resolve, dry-run it and inspect the diff:

cub unit update --space prod --patch --upgrade --dry-run -o mutations backend-clone

-o mutations shows the per-path mutations that the merge would apply, which is usually clearer than diffing the resulting YAML by hand.

Base units

You may choose to maintain base units which you copy to create all of the variants you plan to deploy. Base units are similar to abstract classes in that they aren't intended to be deployed. For example, they need not have attached deployment targets. You can use labels to skip these in bulk applies, or just filter out units without targets (where TargetID IS NOT NULL).

This approach enables a clearer separation of common attributes and custom attributes.

Sample units

A sample unit is similar to a base unit in that it isn't intended to be deployed, but it is meant as a starting point for other units rather than as a mechanism for driving changes to a specific set of variants derived from it.

Placeholders for undetermined values

Base units, and sometimes other new config units, contain default values for most fields, but some may require specific values that cannot yet be provided. This is indicated by placeholders.

A placeholder is a special string or number in the config data which indicates that you must replace it with a real value before the configuration is valid. ConfigHub uses the string "confighubplaceholder" to indicate where a string field needs to be supplied with a value and 9 9s (999999999) for integer fields.

The vet-placeholders function can be installed as a trigger to prevent applying configuration units with unreplaced placeholders, and get-placeholders can return a list of field paths containing placeholders.

Triggers are good for setting general properties of an environment: scale/cost (e.g., replicated vs not, backups vs not), privacy (e.g., public vs not, data with real PII vs not), security (e.g. must be encrypted vs not, can't run as root), and the like.

Links better address more specific resource properties like hostnames, IP addresses, etc.

See managing dependencies for guidance regarding how to replace placeholders with variant-specific values.