Tracking and promoting changes

A ChangeSet names a change within one Space. A rollout is bigger than that: one change, authored once in a base, that has to reach dev, then staging, then every production variant — in that order, and only once the stage before it is actually running it.

Two entities carry that:

A ChangeWorkflow is the definition: the series of stages a change moves through, and what must hold before it may enter each one. It is a pattern, not an execution, and the same one governs every change to a component.
A ChangeOrder is the invocation: one named change moving through that workflow. It is the only identity a change keeps as it crosses a Space boundary, which is what makes "has this change reached prod yet?" a query rather than a diff someone reconstructs.

Neither is required. The upgrade and promote mechanisms work without them. What they add is a name for the change, a declared order for its hops, and an answer to where it has got to.

Declaring a ChangeWorkflow

A ChangeWorkflow is a config Unit written in the Kubernetes Resource Model's shape — an apiVersion, a kind, a name under metadata, everything else under spec. Being KRM, it is an ordinary Kubernetes/YAML Unit, which is the default toolchain, so authoring one needs no flags of its own. Nothing in a cluster reconciles it: the shape is what makes it a Unit like any other, not a resource to deploy.

cub unit create --space acme-workflows acme-main-line - <<'EOF'
apiVersion: confighub.com/v1
kind: ChangeWorkflow
metadata:
  name: acme-main-line
spec:
  stages:
    - name: base
      whereSpace: "Labels.Stage = 'Base'"
    - name: dev
      whereSpace: "Labels.Stage = 'Development'"
    - name: staging
      whereSpace: "Labels.Stage = 'Staging'"
      prerequisites:
        - released
    - name: prod
      whereSpace: "Labels.Stage = 'Production'"
      prerequisites:
        - released
        - healthy
  final:
    prerequisites:
      - released
      - healthy
EOF

Put it somewhere it will not be deployed or cloned. A workflow definition is not part of the component it governs. Keep it in a Space of its own, with no release Target — not in a base Space, where cub variant create would clone it into every variant.

Each stage selects its member Spaces with whereSpace, a where expression over Spaces. Membership is by selector, component and the ChangeOrder's InScopeSpaceIDs, this provides reusability of a workflow across components and a static set of variants for each ChangeOrder. The standard Space labels are what stages select on — Stage, Environment, Region — which cub variant create sets with --stage, --environment, and --region. See managing environments.

The component is not one of them. A stage's selector is conjoined with the component of the Space the ChangeOrder lives in, so the component is declared once, by the change being promoted. That is what lets one definition be cloned to give another component the same shape of rollout, and it is why the base Space must carry a Component label — without one there is nothing to confine the stages to, and that is an error rather than stages selecting every component's Spaces at once.

A whereSpace that names Labels.Component itself is refused:

stage 'staging' names Labels.Component in its whereSpace "Labels.Stage = 'Staging' AND Labels.Component = 'acme'":
the component is the change order's own and is appended to every stage's selector, so remove the predicate

Stating it again either agrees and changes nothing, or disagrees — and a stage that then selects no Space is not obviously wrong: it promotes into nothing and reports nothing. Refusing keeps that failure loud.

Stages are ordered; the Spaces within one are not. A stage is a wave: everything in it may be promoted in any order, or all at once, but the waves are sequential. That is the flexibility a fleet needs — a thousand clusters cannot be a sequence of a thousand steps.

A stage's prerequisites are its entry gates, and they are evaluated over every Space in the stage before it. Two are recognized:

released — that Space has published a Release carrying the change.
healthy — that Space's live status, written back by the GitOps operator, reports Synced / Succeeded / Healthy.

Having taken the change is checked whatever you declare; the prerequisites are checks on top of it. So the base stage above needs no prerequisites — the base has no release Target, releases nothing, and can only be asked whether it has the change. And healthy implies released: live status only exists downstream of a Release.

Live status is read at evaluation time, never cached from when the change was promoted. A stage that was healthy an hour ago and has degraded since does not open the gate ahead of it.

final.prerequisites say when the rollout is done, rather than when a hop may happen. They are evaluated against the last stage — the change has reached it, and it satisfies these checks. That is a reading no promotion makes, there being no hop left to gate once the change is in production.

The definition does not say where the change starts. The base is the Space the ChangeOrder is created in, so it is stated once, by the invocation.

Because it is a Unit, the workflow is versioned, diffable, and restorable like any other configuration — and the revision in force when a ChangeOrder is created is pinned for that ChangeOrder's lifetime, so editing the workflow part way through a rollout cannot change the rules a change already started under.

Naming a change with a ChangeOrder

Make the change in the base Space first, then create the ChangeOrder. This order matters: a ChangeSet has to exist before the revisions that belong to it, but a ChangeOrder is created after the revisions it names, at the moment you decide to promote them.

# 1. Make the change in the base.
cub function set --space acme-base --unit deployment \
    --change-desc "Release v452: bump the image tag" \
    -- set-image-reference-by-uri ghcr.io/acme/acme ":v452"

# 2. Name it, and say which workflow governs where it goes.
cub changeorder create --space acme-base release-v452 \
    --description "Release v452: bump the image tag" \
    --change-workflow acme-workflows/acme-main-line

Creating a ChangeOrder fixes its scope, per Unit of its Space:

Where it is headed comes from the workflow: every Space each stage selects — within the component of the Space being created in — plus that Space itself. Without --change-workflow, pass the list yourself with --in-scope-space (the two flags are mutually exclusive), or leave it empty and wherever its Links reach is where it is headed.
Where each interval starts is the revision the downstream Spaces have already taken — read from the merge cursor of the Links pointing at each Unit. If they disagree, that is an error naming the Unit and the values, rather than a different range being silently promoted to each target.
Where it ends is each Unit's head revision, or the revision a supplied --end-tag marks on it.
It mints two Tags of its own, <slug>-co-start and <slug>-co-end, and stamps the revisions in the interval with the ChangeOrder. The suffixes differ from a ChangeSet's -start/-end so that a ChangeOrder can carry the same name as the ChangeSet whose boundary it adopts.

A ChangeSet is optional. For a single function invocation it adds nothing. Where a change is several edits that must be bounded and locked, open a ChangeSet as described in the change workflow guide, close it, and pass its end Tag:

cub changeorder create --space acme-base release-v452 \
    --description "Release v452" \
    --change-workflow acme-workflows/acme-main-line \
    --end-tag acme-base/release-v452-end

The supplied Tag is read once, to find each Unit's end revision, and never written to. The ChangeOrder still marks with its own, because it marks Units the ChangeSet never touched.

A Unit the change carries nothing for is marked, not dropped. Its start and end Tags land on the same revision: this is where the ChangeOrder applied, and none of this Unit's revisions belong to it. That is what lets cub release publish --revision ChangeOrder:<slug> pin every Unit the ChangeOrder covered rather than falling back to the head of each one the change did not touch.

cub changeorder get reports Skipped Units — the Units of its own Space it carries no revisions of, and why of each — so a ChangeOrder narrower than you expected says so instead of being silently narrow.

In a release pipeline, this is two commands after the build: set the new image reference in the base, then create the ChangeOrder that names the release. Check first that a ChangeOrder of that name does not already exist — creating one is what fixes the range, so a second run of the same version should stop rather than bump the image and then fail on the slug:

if cub changeorder get --space acme-base "release-$VERSION" >/dev/null 2>&1 ; then
    echo "$VERSION has already been released" >&2
    exit 1
fi

Do not use --allow-exists here. It returns the ChangeOrder already there, whose end Tag is on the head as it was then; if the unit has moved since, a later promotion carrying that ChangeOrder walks a range that stops short of this release.

Promoting stage to stage

cub variant promote moves the change. With --change-order and no Space, it advances the change into the first stage it has not reached:

cub variant promote --change-order acme-base/release-v452

To name the stage explicitly:

cub variant promote --change-order acme-base/release-v452 --target-stage staging

Preview first. --dry-run reports the Units that would be upgraded and added without changing anything; -o mutations shows the values that would change:

cub variant promote --change-order acme-base/release-v452 --target-stage staging --dry-run -o mutations

To promote one Space rather than a whole stage — repairing a variant that failed inside a partial stage, or a ChangeOrder with no workflow at all — name it:

cub variant promote acme-staging-euw1 --change-order acme-base/release-v452

What promotion does, per target Space:

Clones the Units the target does not have yet, at the revision the ChangeOrder starts from — before upgrading, which is the reverse of a plain promote. The change then replays into the clone, so it ends up carrying the revisions the change made everywhere else rather than arriving whole with the change already folded in. A Unit created upstream after the ChangeOrder was fixed is outside the change: it is reported rather than cloned, and a promotion without --change-order is what adds it.
Upgrades the Units that are behind, with the ChangeOrder supplying the range so the merge stops where the change ends rather than at the upstream's head. A change made upstream after the ChangeOrder was created is left behind.
Marks what it did. The ChangeOrder's start Tag goes on the target's head before the merge and its end Tag on the revision the merge arrives at, whether the range landed as one revision or as one per source revision.

Promotion moves configuration only. It does not publish a Release.

The gates are the stage's, not any one Space's. They are evaluated once over the whole membership of the stage ahead. Naming a later stage does not skip what precedes it: --target-stage prod while staging is unsatisfied is refused, naming the prerequisite and the Space that blocks it.

A stage can land partway. Its Spaces are promoted one at a time; one that fails is reported and the ones after it are still promoted, and the result says how many landed. Running the promotion again is what repairs it: a Space that already took the change is passed over, because a ChangeOrder reaches a Unit once and only once — a Unit already carrying the end Tag has taken the change, whatever has happened to it since.

Enforcement is client-side in this phase. cub variant promote checks the prerequisites; a caller that goes straight to cub unit update --upgrade --change-order bypasses them. That path is not equivalent to a promotion anyway — it clones no missing Units and copies no Links — but it does move the change past the gates.

Releasing each stage

Publishing is a separate operation, never a step of promote. A stage that has been promoted but not released is a legitimate resting state — and the released prerequisite is what makes the next stage wait for it.

Approve, then publish, per Space:

cub variant approve acme-staging-euw1
cub release publish --revision ChangeOrder:acme-base/release-v452 acme-staging-euw1

--revision ChangeOrder:<slug> pins each Unit to the revision the change arrived at there, rather than to its head, so a Release describes the change and not whatever else has happened since. Before:ChangeOrder:<slug> names the state before it, which is how the previous state is re-released to roll the change back.

cub release publish takes one Space at a time; releasing a whole stage is that command across the stage's Spaces. See Publishing Releases.

Where has it got to

Nothing writes "current stage" anywhere. A ChangeOrder's position is derived when it is read, from the Tags its promotions left and the Releases that have been published — which is why it cannot go stale, and why a stage that has degraded since it was promoted reads as degraded now.

cub changeorder get --space acme-base release-v452

reports:

Field	What it says
`State`	`New`, `InProgress`, `Resolved`, `Released` — or `Aborted`, `Restored`, `RestoreReleased`
`Stage`	the last stage of the governing workflow the change has reached
`Completed`	whether `final.prerequisites` hold against the last stage
`In-Scope Spaces`	where it is headed
`Resolved Spaces`	where it has been fully promoted
`Released Spaces`	where a Release carrying it has been published
`Skipped Units`	the Units of its own Space it carries no revisions of, and why

State and Completed are different readings and can legitimately disagree. State never consults live status, so a workflow whose final.prerequisites name healthy is not complete at Released; and State reduces over every Space in scope, so a last stage that is released and healthy is complete while an earlier stage that was never gated on released leaves State at Resolved.

All of the Space sets are queryable, so fleet questions are one command:

# Which rollouts are still in flight?
cub changeorder list --space '*' --where "State = 'InProgress'"

# Has this change reached that Space?
cub changeorder list --space acme-base --where "ResolvedSpaceIDs ? '<space-id>'"

# Which have got past their own base?
cub changeorder list --space acme-base --where "LEN(ResolvedSpaceIDs) > 1"

Revisions carry the ChangeOrder too, upstream and downstream alike, so "what did this change actually do here?" is a revision query in any Space it reached.

Undoing a rollout: abort and demote

Aborting a ChangeOrder is the record that the change is not coming to the Spaces still waiting for it. Set a reason:

cub changeorder update --space acme-base release-v452 \
    --aborted-reason "superseded by v453"

Aborting changes nothing about the Spaces that already took the change — which is exactly the state that needs undoing. cub variant demote is what takes it back out, one Space at a time:

cub variant demote acme-prod-use1 --change-order acme-base/release-v452 \
    --change-desc "back out v452"

Each Unit the ChangeOrder marked there is restored to the revision it was at before the change — one step, however many revisions the promotion made, because the promotion's start Tag marks that state by name.

Preview it first. A Unit whose head has moved past where the change ended has later changes the restore will drop, and --dry-run names them before anything is written:

cub variant demote acme-prod-use1 --change-order acme-base/release-v452 --dry-run

Things worth knowing:

Demote requires the abort. Undoing a ChangeOrder nobody has said is not coming is a race with whoever is still promoting it.
Which Units are restored is the ChangeOrder's answer, not a --where of yours: the Units of that Space its start Tag marks. A Unit it never marked is an error rather than a Unit passed over.
It is idempotent, like promotion. A Unit already carrying the restore Tag is left alone, so a demote that landed partway is run again to reach the Units it did not — and a Unit edited forward since it was undone keeps that work.
A ChangeOrder travels one way. An aborted one is refused by every promotion, and its reason cannot be cleared once anything has been restored. A change to promote after an undoing is a new ChangeOrder.
The ChangeOrder's own Tags stay where they are. A Space that took the change took it, whatever has happened since. The restore Tag is what says it was taken back out again, and State reconciles the two: Restored once every Space that took the change has been demoted, RestoreReleased once every Space that had released it has released the restored revisions.
Nothing is promoted or released for you. Each Space is demoted on its own account, and publishing the restored revisions is a separate cub release publish — because what a Space goes back to has to be released where the change was released. Re-applying anything the restore dropped is a forward change to make afterwards.

To undo a change that was never named by a ChangeOrder, or to take out one range of revisions inside a single Space, revert it instead.

Change Workflow — making and releasing a change within one variant
Creating and managing variants — clone, upgrade, protection, conflicts
Managing environments — the Space labels stages select on
Publishing Releases — the Release command reference
Advanced merging techniques — what a promotion's merge does