Skip to content

Advanced merging techniques

Everything in this guide is optional. Cloning a space, promoting it, and protecting the values a variant chose for itself — the material in creating and managing variants — covers the ordinary path. This one describes what the merge engine does underneath, and the controls for the cases where the defaults are not what you want.

--upgrade, --resolve against an UpgradeUnit/MergeUnits Link, --merge-source, and --merge-external-source all share that engine.

How merging works

A merge walks the range it is given. It takes the source's revisions in order, and where a revision was produced by a function it re-runs that function against the target unit rather than copying the paths the function happened to touch upstream. See Replay. Each source revision that has an effect downstream becomes a downstream revision of its own, so the variant's history says which upstream change produced each of its own. --squash merges the range as one rebased diff in one revision instead. Since a walked merge can leave several revisions, cub unit update --restore -1 afterwards undoes only its last step; to have the whole merge to undo, make it within a ChangeSet and restore to Before:ChangeSet:<slug>, or promote a ChangeOrder and demote it.

Each step of the walk — and the whole range, when squashed — is a four-way merge:

  1. Diff the source between two source revisions — for upgrade and MergeUnits, that range is the Link's UpstreamLastMergedRevisionNum to the upstream unit's current HeadRevisionNum; for --merge-source it's --merge-base to --merge-end; for --merge-external-source it's the previously-recorded external file (or the revision selected by --merge-base) to the new external file.
  2. Diff the downstream unit between the previously-merged downstream revision and its current head, so that the downstream's existing local differences from the source baseline are recognized.
  3. Subtract any changes in (1) that conflict with the downstream's local differences from (2). This step is off by default — see Enabling the subtraction step.
  4. Apply the remaining changes as a patch to the downstream unit's head.

What keeps a merge from overwriting a downstream customization, by default, is not step (3) but the protection recorded per path on the downstream unit. Two other mechanisms can be turned on per merge or per Link — the subtraction in step (3), and a WhereMutation filter — and either is the way to preserve downstream differences without marking them one at a time. WhereMutation adds to the stored flags; subtraction replaces them.

UpstreamRevisionNum on the unit and UpstreamLastMergedRevisionNum on the Link are updated when the merge completes — a step at a time as the range is walked. UpstreamLastMergedRevisionNum can be edited if you need to advance or rewind the merge cursor without doing a real merge.

Replay: re-running upstream changes here

A patch carries the paths the upstream recorded. Replay carries the change: for each upstream revision in the range, the function invocations it recorded are re-run against this unit, and the result is applied. The difference shows up whenever the downstream is not shaped exactly like the upstream:

# Upstream has one container; the deployment here has two, because this
# variant added a metrics sidecar.
cub function set --space app-base --unit web set-container-probe-defaults

# Upgrading re-runs set-container-probe-defaults here, so it reaches both
# containers -- including the one upstream has never seen.
cub unit update --space app-prod web --patch --upgrade

A rebased patch could only have carried probes for the container the upstream had. Replay is what makes an upstream policy change apply to what a variant added.

Not everything can be re-run. A revision that was a hand edit, an import, or another merge records no invocation to replay, and some functions are excluded (a path with a hardcoded index, a function that has since been removed, one that ran on a Channel). Those revisions are merged as a patch, exactly as before, and what happened to each step is recorded on the resulting mutation:

cub mutation list --space app-prod web --select "*"     -o jq='[.[] | .Mutation | {num: .MutationNum, outcome: .ReplayOutcome, why: .ReplayReason}]'

Replayed, ReplayedNoEffect (re-run, matched nothing here), NotReplayable, ReplayUnavailable, ReplayFailed, or Patched. A squashed merge records none of them, which is how you tell a merge that never considered replay from one that considered it and declined.

An invocation that will be replayed across several variants has to select what it changes by name rather than by position, since variants differ in collection ordering and contents. See writing changes that generalize across variants.

Squashing a range

--squash merges the range as one rebased diff and records it as one revision — what a merge did before replay existed:

cub unit update --space app-prod web --patch --upgrade --squash

It is valid with --upgrade, --merge-source, and --resolve, and rejected elsewhere, since nothing else has a range to walk. A Link can ask for it standingly, which is what to do when a Link should always merge this way:

cub link update --space app-prod --patch upgrade-web --squash

Reasons to squash:

  • The upstream's revision history is noise in this variant, and one revision per promotion reads better than one per upstream change.
  • The Link relies on the subtraction step to preserve downstream differences. A replayed step re-runs against this unit's current data, so there is no baseline to subtract a difference from — subtraction only applies to a squashed merge. If a Link preserves overrides that way rather than with protection, give it --squash.
  • More than one Link is being resolved at once. --resolve Link:* over a unit that has both a merge Link and a needs/provides Link squashes the merge automatically, because replay needs the merge to be the whole of the update. Resolving the merge Link by itself walks its range.

Note that --squash is a request-time or Link-level choice, not a property of the units: the same Link can be squashed on one promotion and walked on the next.

MutationSources: which mutation last touched each path

Every time a unit's data changes, ConfigHub records which mutation last set each configuration path on the unit's MutationSources. That includes hand edits, function invocations (whether ad hoc, triggered, or part of a clone or upgrade), needs/provides resolutions, and merge results. MutationSources is what lets the merge engine tell "the upstream changed this field" apart from "the downstream changed this field locally," and it is where each path's protection is stored.

A few consequences worth knowing:

  • Restore preserves provenance. cub unit update --restore rewinds both the unit's Data and its MutationSources to the chosen revision, so subsequent merges treat the restored state as the truth about who-touched-what.
  • You can "reset" MutationSources by removing all data and adding it back. Updating a unit to empty config and then back to the desired config replaces every entry in MutationSources with the new mutation. This is the supported way to drop accumulated provenance when you want a future upgrade to overwrite paths it would otherwise consider local overrides.
  • Setting a field to the value the upstream already has is not an override, as far as the subtraction step is concerned: it looks identical to the merged-in upstream value, so there is nothing to subtract. With subtraction off, whether that path is protected depends on whether you asked for it.

See the Mutation Sources concept for the full picture.

WhereMutation: protecting downstream paths by rule

WhereMutation on a Link (or --where-mutation on the operation) is a filter over the downstream unit's mutation history that selects which downstream paths the merge must not overwrite. It is empty by default on every Link, including the UpgradeUnit Link a clone gets.

It is unioned with the stored Protected flags rather than replacing them, so it can only ever protect more. A path you protected with cub unit set-protection stays protected whatever the filter says, and a filter that matches nothing behaves exactly like no filter at all. Use it when you want a policy expressed as one rule over provenance rather than path by path.

For example, to protect everything the downstream authored — hand edits, function invocations, needs/provides bindings, trigger output — while leaving content that arrived through the clone lineage the upgrade's to update:

cub link update --space prod upgrade-backend \
    --where-mutation "Revision.Source NOT IN ('CloneUnit', 'UpgradeUnit', 'MergeUnits', 'MergeExternal', 'ApplyConflicts')"

To clear it again and go back to the stored protection alone:

cub link update --space prod --patch --where-mutation="" upgrade-backend

Any Mutation field can drive the rule, so a Link whose WhereMutation selects the mutations that set spec.replicas will never overwrite the downstream's replica count.

Enabling the subtraction step

The subtraction in step (3) of How merging works is off by default. Turning it on adds a second rule on top of the source patch: anything the downstream unit differs from the merge base in is dropped from the patch, whatever the stored Protected flags say about those paths.

  • Per request, with --merge-enable-subtraction on cub unit update / cub unit update --patch.
  • Per Link, with cub link create/cub link update --merge-enable-subtraction (the Link's MergeEnableSubtraction field), which applies whenever that Link is resolved (including via --upgrade).

Reach for it when the downstream's differences from the base are all deliberate and you would rather not reason about them path by path. Note that it infers overrides from the diff at merge time, so it has no memory: if you change a downstream field back to the value the upstream has, it is no longer a difference and a later upstream change to that field will propagate. The stored protection does remember.

Two cases worth knowing:

  • A self merge (--merge-source Self) always runs with subtraction off, regardless of the flag, since it is reapplying a range of the unit's own changes. It ignores the stored protection too, for the same reason.
  • Subtraction treats every difference from the merge base as an override, so avoid it when the base is not a true common ancestor of the source and target — merging between independent units, say — where it would drop most of the patch.

WhereResource: scoping which upstream resources participate

WhereResource on a Link (or --where-resource on the operation) filters the source side: only resources matching the expression contribute to the merge. It is useful for splitting a single upstream unit across multiple downstream units (e.g., separating CustomResourceDefinitions from the rest), or for narrowing an upgrade to a subset of resources.

Strategic merge patch and merge keys

ConfigHub treats configuration as data, not as text, so it merges resource-by-resource and field-by-field — independent edits to different fields of the same resource don't conflict. For Kubernetes resources, ConfigHub uses its own implementation of strategic merge patch, which is aware of Kubernetes merge keys (the x-kubernetes-patch-merge-key markers on associative lists like containers, volumes, env, and ports).

This has a few important implications:

  • Multi-line string fields are merged as text, except for embedded JSON. Free-form text (like a shell script in a ConfigMap value) is merged line-by-line rather than character-by-character. ConfigHub auto-detects JSON-valued strings and merges them as structured data instead. For configuration files that you want managed as structured data — properties, env, INI, TOML, YAML, JSON — store them in their own units rather than embedding them as strings in a ConfigMap; see application configuration for the recommended pattern.
  • Renames look like delete + add. Changing the name of a resource, or the merge key of a list element (e.g., a container's name), looks to the merge engine like a deletion of the old element followed by addition of a new one. ConfigHub attempts to detect renames via a similarity heuristic — if the bodies match closely, it carries the renamed element forward in place and preserves any independent downstream overrides on it. If they differ enough to fall outside the heuristic, the downstream sees a delete + add and any downstream overrides on the deleted element are lost.
  • Lists without a merge key are matched by position, but structural changes are recognized. For a list ConfigHub has no merge key for — a CRD's own arrays, a list of strings — elements correspond by index. When upstream removes or inserts an element, ConfigHub records that removal or insertion rather than treating every element after it as changed, so downstream customizations on the surrounding elements survive. Two elements that have nothing in common are read as a removal plus an insertion rather than as an edit of one into the other, and a downstream customization on a removed element goes with it. An element that the downstream both edited and moved can't be found this way; name it with a comment instead.
  • Reordering may not do what you expect. ConfigHub tries to preserve the order of merge-keyed list elements, but reordering elements both upstream and downstream can produce surprising merge results. If you care about ordering, change it on one side at a time.

If you're unsure how a merge will resolve, dry-run it and inspect the diff:

cub unit update --space prod --patch --upgrade --dry-run -o mutations backend-clone

-o mutations shows the per-path mutations that the merge would apply, which is usually clearer than diffing the resulting YAML by hand.

Naming list elements with confighub:id

For a list without a merge key, ConfigHub identifies each element by a digest of its content, with its position as the fallback. That breaks down when the downstream has edited an element and moved it: the digest no longer matches, and the position now holds a different element, so an upstream change to that element lands on the wrong one. A Traefik IngressRoute is the usual example. Its routes have no merge key, and the field that looks identifying, match, is the one a variant typically rewrites.

A structured comment, # confighub:id, says which element is which. It plays the role a merge key plays for containers, but is declared in the configuration rather than in the resource's schema:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: web
spec:
  routes:
  # confighub:id=api
  - match: Host(`api.example.com`)
    kind: Rule
    priority: 100
    services:
    - name: svc-api
      port: 80
  # confighub:id=admin
  - match: Host(`admin.example.com`)
    kind: Rule
    priority: 50
    services:
    - name: svc-admin
      port: 80

There are two forms:

  • confighub:id=<value> gives the element the identity <value>. The comment can go on the line above the element, inline after a scalar element (- --verbose # confighub:id=verbose), or inline on any of the element's fields, which is where a comment at the end of a mapping element's first line lands.
  • A bare confighub:id on a field makes that field's value the identity, which is how you nominate a field as a key for a list that doesn't have one. Choose a field the variants don't change, since the identity changes when the value does:

    users:
    - name: admin   # confighub:id
      role: owner
    

With the comments in place, a variant can rewrite both hostnames and swap the two routes, and an upstream change to the admin route still reaches the admin route:

cub function set --space app-base --unit routes --change-desc "Raise the admin route priority" \
    set-int-path traefik.io/v1alpha1/IngressRoute spec.routes.1.priority 75
cub unit update --space app-prod routes --patch --upgrade

The path names the route by position, so the merge applies this change as a patch rather than replaying it, and the identity the patch recorded for that position is what finds the route downstream. The downstream's admin route gets priority 75, its api route keeps 100, and the downstream's own match values are untouched. The comments survive the merge, so they keep working on later upgrades.

Things to know:

  • Both sides need the comment. The upstream records the identity in the paths of its changes, and the downstream's elements are matched against it. Write it in the upstream unit, where clones inherit it, rather than only in the variant that needed it.
  • The comment takes precedence over everything ConfigHub infers, including the content digest and the position.
  • An identity has to be unique within its list. If two elements carry the same identity, it identifies neither, and both are matched by position like the rest.
  • An identity containing ,, =, or ; is ignored, since those characters are part of path syntax.
  • Command-line flags are identified automatically. An element of a list of strings that holds a flag with an inline value, such as --log.level=INFO, is identified by the flag name, so --log.level=INFO and --log.level=DEBUG are the same element, and an upstream change still finds it after the downstream has changed its value and moved it. A flag that appears more than once in the list, such as Helm's --set, is not unique, and falls back to position as above. A confighub:id comment on the element takes precedence over the flag name.