Skip to content

Advanced merging techniques

Everything in this guide is optional. Cloning a space, promoting it, and protecting the values a variant chose for itself — the material in creating and managing variants — covers the ordinary path. This one describes what the merge engine does underneath, and the controls for the cases where the defaults are not what you want.

--upgrade, --resolve against an UpgradeUnit/MergeUnits Link, --merge-source, and --merge-external-source all share that engine.

How merging works

A merge walks the range it is given. It takes the source's revisions in order, and where a revision was produced by a function it re-runs that function against the target unit rather than copying the paths the function happened to touch upstream. See Replay. Each source revision that has an effect downstream becomes a downstream revision of its own, so the variant's history says which upstream change produced each of its own. --squash merges the range as one rebased diff in one revision instead.

Each step of the walk — and the whole range, when squashed — is a four-way merge:

  1. Diff the source between two source revisions — for upgrade and MergeUnits, that range is the Link's UpstreamLastMergedRevisionNum to the upstream unit's current HeadRevisionNum; for --merge-source it's --merge-base to --merge-end; for --merge-external-source it's the previously-recorded external file (or the revision selected by --merge-base) to the new external file.
  2. Diff the downstream unit between the previously-merged downstream revision and its current head, so that the downstream's existing local differences from the source baseline are recognized.
  3. Subtract any changes in (1) that conflict with the downstream's local differences from (2). This step is off by default — see Enabling the subtraction step.
  4. Apply the remaining changes as a patch to the downstream unit's head.

What keeps a merge from overwriting a downstream customization, by default, is not step (3) but the protection recorded per path on the downstream unit. Two other mechanisms can be turned on per merge or per Link — the subtraction in step (3), and a WhereMutation filter — and both decide in place of the stored flags rather than alongside them; either is the way to preserve downstream differences without marking them one at a time.

UpstreamRevisionNum on the unit and UpstreamLastMergedRevisionNum on the Link are updated when the merge completes — a step at a time as the range is walked. UpstreamLastMergedRevisionNum can be edited if you need to advance or rewind the merge cursor without doing a real merge.

Replay: re-running upstream changes here

A patch carries the paths the upstream recorded. Replay carries the change: for each upstream revision in the range, the function invocations it recorded are re-run against this unit, and the result is applied. The difference shows up whenever the downstream is not shaped exactly like the upstream:

# Upstream has one container; the deployment here has two, because this
# variant added a metrics sidecar.
cub function set --space app-base --unit web set-container-probe-defaults

# Upgrading re-runs set-container-probe-defaults here, so it reaches both
# containers -- including the one upstream has never seen.
cub unit update --space app-prod web --patch --upgrade

A rebased patch could only have carried probes for the container the upstream had. Replay is what makes an upstream policy change apply to what a variant added.

Not everything can be re-run. A revision that was a hand edit, an import, or another merge records no invocation to replay, and some functions are excluded (a path with a hardcoded index, a function that has since been removed, one that ran on a worker). Those revisions are merged as a patch, exactly as before, and what happened to each step is recorded on the resulting mutation:

cub mutation list --space app-prod web --select "*"     -o jq='[.[] | .Mutation | {num: .MutationNum, outcome: .ReplayOutcome, why: .ReplayReason}]'

Replayed, ReplayedNoEffect (re-run, matched nothing here), NotReplayable, ReplayUnavailable, ReplayFailed, or Patched. A squashed merge records none of them, which is how you tell a merge that never considered replay from one that considered it and declined.

An invocation that will be replayed across several variants has to select what it changes by name rather than by position, since variants differ in collection ordering and contents. See writing changes that generalize across variants.

Squashing a range

--squash merges the range as one rebased diff and records it as one revision — what a merge did before replay existed:

cub unit update --space app-prod web --patch --upgrade --squash

It is valid with --upgrade, --merge-source, and --resolve, and rejected elsewhere, since nothing else has a range to walk. A Link can ask for it standingly, which is what to do when a Link should always merge this way:

cub link update --space app-prod --patch upgrade-web --squash

Reasons to squash:

  • The upstream's revision history is noise in this variant, and one revision per promotion reads better than one per upstream change.
  • The Link relies on the subtraction step to preserve downstream differences. A replayed step re-runs against this unit's current data, so there is no baseline to subtract a difference from — subtraction only applies to a squashed merge. If a Link preserves overrides that way rather than with protection, give it --squash.
  • More than one Link is being resolved at once. --resolve Link:* over a unit that has both a merge Link and a needs/provides Link squashes the merge automatically, because replay needs the merge to be the whole of the update. Resolving the merge Link by itself walks its range.

Note that --squash is a request-time or Link-level choice, not a property of the units: the same Link can be squashed on one promotion and walked on the next.

MutationSources: which mutation last touched each path

Every time a unit's data changes, ConfigHub records which mutation last set each configuration path on the unit's MutationSources. That includes hand edits, function invocations (whether ad hoc, triggered, or part of a clone or upgrade), needs/provides resolutions, and merge results. MutationSources is what lets the merge engine tell "the upstream changed this field" apart from "the downstream changed this field locally," and it is where each path's protection is stored.

A few consequences worth knowing:

  • Restore preserves provenance. cub unit update --restore rewinds both the unit's Data and its MutationSources to the chosen revision, so subsequent merges treat the restored state as the truth about who-touched-what.
  • You can "reset" MutationSources by removing all data and adding it back. Updating a unit to empty config and then back to the desired config replaces every entry in MutationSources with the new mutation. This is the supported way to drop accumulated provenance when you want a future upgrade to overwrite paths it would otherwise consider local overrides.
  • Setting a field to the value the upstream already has is not an override, as far as the subtraction step is concerned: it looks identical to the merged-in upstream value, so there is nothing to subtract. With subtraction off, whether that path is protected depends on whether you asked for it.

See the Mutation Sources concept for the full picture.

WhereMutation: scoping which downstream paths can be overwritten

WhereMutation on a Link (or --where-mutation on the operation) is a filter over the downstream unit's mutation history that selects which downstream paths are eligible to be overwritten by the source-side changes. It is empty by default on every Link, including the UpgradeUnit Link a clone gets.

Setting one replaces the stored Protected flags for that merge: eligibility is re-derived from mutation history at merge time, and any per-path decision you made with cub unit set-protection no longer applies. Use it when you want the policy expressed as one rule over provenance rather than path by path. For example, to preserve only what did not come from a clone, upgrade, or merge — the rule upgrade Links used to be created with:

cub link update --space prod upgrade-backend \
    --where-mutation "Revision.Source IN ('CloneUnit', 'UpgradeUnit', 'MergeUnits')"

To clear it again and go back to the stored protection:

cub link update --space prod --patch --where-mutation="" upgrade-backend

WhereMutation can also be used proactively to "protect" parts of a unit from upgrade — for example, a Link whose WhereMutation excludes mutations under spec.replicas will never overwrite the downstream's replica count.

Enabling the subtraction step

The subtraction in step (3) of How merging works is off by default. Turning it on adds a second rule on top of the source patch: anything the downstream unit differs from the merge base in is dropped from the patch, whatever the stored Protected flags say about those paths.

  • Per request, with --merge-enable-subtraction on cub unit update / cub unit update --patch.
  • Per Link, with cub link create/cub link update --merge-enable-subtraction (the Link's MergeEnableSubtraction field), which applies whenever that Link is resolved (including via --upgrade).

Reach for it when the downstream's differences from the base are all deliberate and you would rather not reason about them path by path. Note that it infers overrides from the diff at merge time, so it has no memory: if you change a downstream field back to the value the upstream has, it is no longer a difference and a later upstream change to that field will propagate. The stored protection does remember.

Two cases worth knowing:

  • A self merge (--merge-source Self) always runs with subtraction off, regardless of the flag, since it is reapplying a range of the unit's own changes. It ignores the stored protection too, for the same reason.
  • Subtraction treats every difference from the merge base as an override, so avoid it when the base is not a true common ancestor of the source and target — merging between independent units, say — where it would drop most of the patch.

WhereResource: scoping which upstream resources participate

WhereResource on a Link (or --where-resource on the operation) filters the source side: only resources matching the expression contribute to the merge. It is useful for splitting a single upstream unit across multiple downstream units (e.g., separating CustomResourceDefinitions from the rest), or for narrowing an upgrade to a subset of resources.

Strategic merge patch and merge keys

ConfigHub treats configuration as data, not as text, so it merges resource-by-resource and field-by-field — independent edits to different fields of the same resource don't conflict. For Kubernetes resources, ConfigHub uses its own implementation of strategic merge patch, which is aware of Kubernetes merge keys (the x-kubernetes-patch-merge-key markers on associative lists like containers, volumes, env, and ports).

This has a few important implications:

  • Multi-line string fields are merged as text, except for embedded JSON. Free-form text (like a shell script in a ConfigMap value) is merged line-by-line rather than character-by-character. ConfigHub auto-detects JSON-valued strings and merges them as structured data instead. For configuration files that you want managed as structured data — properties, env, INI, TOML, YAML, JSON — store them in their own units rather than embedding them as strings in a ConfigMap; see application configuration for the recommended pattern.
  • Renames look like delete + add. Changing the name of a resource, or the merge key of a list element (e.g., a container's name), looks to the merge engine like a deletion of the old element followed by addition of a new one. ConfigHub attempts to detect renames via a similarity heuristic — if the bodies match closely, it carries the renamed element forward in place and preserves any independent downstream overrides on it. If they differ enough to fall outside the heuristic, the downstream sees a delete + add and any downstream overrides on the deleted element are lost.
  • Lists without a merge key are matched by position, but structural changes are recognized. For a list ConfigHub has no merge key for — a CRD's own arrays, a list of strings — elements correspond by index. When upstream removes or inserts an element, ConfigHub records that removal or insertion rather than treating every element after it as changed, so downstream customizations on the surrounding elements survive. Two elements that have nothing in common are read as a removal plus an insertion rather than as an edit of one into the other, and a downstream customization on a removed element goes with it.
  • Reordering may not do what you expect. ConfigHub tries to preserve the order of merge-keyed list elements, but reordering elements both upstream and downstream can produce surprising merge results. If you care about ordering, change it on one side at a time.

If you're unsure how a merge will resolve, dry-run it and inspect the diff:

cub unit update --space prod --patch --upgrade --dry-run -o mutations backend-clone

-o mutations shows the per-path mutations that the merge would apply, which is usually clearer than diffing the resulting YAML by hand.