Tracking and promoting changes

A ChangeSet bounds a change where it is made, in the Spaces whose units it is open on. A rollout is bigger than that: one change, authored once in a base, that has to reach dev, then staging, then every production variant — in that order, and only once the stage before it is actually running it.

Two entities carry that:

• A ChangeWorkflow is the definition: the series of stages a change moves through, and what must hold before it may enter each one or be released in it. It is a pattern, not an execution, and the same one governs every change to a component.
• A ChangeOrder is the invocation: one named change moving through that workflow. It is the only identity a change keeps as it crosses a Space boundary, which is what makes "has this change reached prod yet?" a query rather than a diff someone reconstructs.

Neither is required. The upgrade and promote mechanisms work without them. What they add is a name for the change, a declared order for its hops, and an answer to where it has got to.

Declaring a ChangeWorkflow

A ChangeWorkflow is an entity in a Space, created with cub changeworkflow create. Keep workflows in a Space of their own, with no release Target: a workflow is not part of the component it governs, and one in a base Space would be cloned into every variant by cub variant create.

The whole workflow can be written in a file, in the fields cub changeworkflow get -o json prints:

cub space create acme-workflows
cub changeworkflow create --space acme-workflows acme-main-line --filename - <<'EOF'
Stages:
  - Name: dev
    WhereSpace: "Labels.Stage = 'Development'"
  - Name: staging
    WhereSpace: "Labels.Stage = 'Staging'"
    Prerequisites: [Validated, Released]
  - Name: prod
    WhereSpace: "Labels.Stage = 'Production'"
    Prerequisites: [Released, Healthy, not-frozen]
Final:
  Prerequisites: [Released, Healthy]
CustomPrerequisites:
  - Name: not-frozen
    Expression: "cel:!('code-freeze' in Space.Annotations) || Space.Annotations['code-freeze'] != 'true'"
    Description: the space is not under a code freeze
EOF

A workflow whose Stages are named after the Stage label values they select, and that gates every Stage after the first alike, can be written with flags instead. Each --stage selects Labels.Stage = '<name>', and --prerequisites applies to every Stage after the first and to Final. The first Stage's gates are evaluated over the base, which a set meant for the hops between Stages does not fit, so a first Stage with gates of its own is written in a file:

cub changeworkflow create --space acme-workflows acme-simple \
    --stage Development --stage Staging --stage Production \
    --prerequisites Released,Healthy

Each field of a Stage has a flag of its own, given as <stage>=<value>, and so do the prerequisites a workflow declares. To change one Stage of an existing workflow and leave the rest as it is:

cub changeworkflow update --space acme-workflows acme-simple \
    --stage-where-space "Production=Labels.Stage = 'Production' AND Labels.Region = 'us-east1'" \
    --stage-prerequisites 'Production=Released;Healthy;two-approvers' \
    --attestation-prerequisite two-approvers --attestation-prerequisite-count two-approvers=2

On an update, the Stages named with --stage are the Stages the workflow has, in that order: one that is named again keeps its other fields, and one that isn't named is removed.

Each Stage selects its member Spaces with WhereSpace, a where expression over Spaces. The standard Space labels are what Stages select on — Stage, Environment, Region — which cub variant create and cub variant upload set with --stage, --environment, and --region. Set --stage on every variant a workflow will promote into: a Space with no Stage label is in no Stage that selects on it. See managing environments.

Where the rollout goes is the ChangeOrder's to say, not the workflow's. A Stage's members are the Spaces its WhereSpace selects among the Spaces its ChangeOrder is headed for, its InScopeSpaceIDs (below). Nothing else is implied, the component included, so one workflow can govern the rollouts of several components unchanged, each ChangeOrder bringing its own scope. A WhereSpace can name a component, as ComponentID = '<component-id>', but that ties the workflow to that one component; selecting the component is usually the ChangeOrder's job, with cub changeorder create --component.

Stages are ordered; the Spaces within one are not. A Stage is a wave: everything in it may be promoted in any order, or all at once, but the waves are sequential. That is the flexibility a fleet needs — a thousand clusters cannot be a sequence of a thousand steps.

The definition does not say where the change starts. The base is the Space the ChangeOrder is created in, so it is stated once, by the invocation. The first Stage's prerequisites are evaluated over the base, which is how a gate is put on the change before it leaves it. A first Stage that selects only the base Space, such as Labels.Stage = 'Base', with the Spaces downstream of it in the Stages after, is how a base with a release Target has its own ReleasePrerequisites, and the second Stage can then require that the base released the change.

Prerequisites

A Stage's Prerequisites are its entry gates, and they are evaluated over every Space in the Stage before it, and the first Stage's over the base. A ChangeOrder of type Invoke or a fan-out type refuses a workflow that gives its first Stage prerequisites, since its change is in no Space before it is promoted into one. Having taken the change is checked whatever you declare; the prerequisites are checks on top of it. Three are built in:

• Validated — no Unit the change changed in that Space has Validation Errors on what a Release of the change would bundle there. It needs no Release, so it also works for Spaces that are never released.
• Released — that Space has published a Release carrying the change. A Space with no release Target has nothing to release, so it passes, as it does not hold back the ChangeOrder's Released state; it is not counted among the Spaces that released the change.
• Healthy — that Space has published a Release carrying the change, and the live status of its latest published Release, written back by the GitOps operator, reports Synced and Healthy with no operation Running or Failed. It implies Released, and a Space with no release Target can never satisfy it, since nothing is deployed from it.

Live status belongs to a Release, so a Release the operator has not reported on yet is not healthy, however healthy the one before it was: a change promoted into a Stage just after it was released there waits for that Release to be reported on. It is read at evaluation time, never cached from when the change was promoted, so a Stage that was healthy an hour ago and has degraded since does not open the gate after it either.

Custom prerequisites are declared once under CustomPrerequisites and named wherever they apply. Each is a CEL expression, prefixed with cel:, over the Space being evaluated, the ChangeOrder, and the Release of that Space carrying the change (the one the ChangeOrder's Releases names), each as the JSON the API returns. Anything that can write an annotation on a Space or a Release — a code freeze switch, a load test, a deployment verification tool — can gate a rollout this way. Expressions are compiled when the workflow is written, so a broken one is refused then rather than holding up a promotion.

Attestation prerequisites require approvals and other Attestations, declared under AttestationPrerequisites. A Stage names them in Prerequisites, to gate entry, or in ReleasePrerequisites, to gate publishing a Release in the Stage's own Spaces. See review and approval.

Final.Prerequisites say when the rollout is done, rather than when a hop may happen. They are evaluated against the last Stage — the change has reached it, and it satisfies these checks. That is a reading no promotion makes, there being no hop left to gate once the change is in production.

Changing a workflow

cub changeworkflow update edits a workflow in place. A ChangeOrder takes a copy of its workflow's definition when it is created and is judged against that copy for its whole life, so editing a workflow part way through a rollout cannot change the rules a change already started under. The edit applies to ChangeOrders created after it. To move a change under the new rules, create a new ChangeOrder for it.

cub changeworkflow create --changeworkflow <slug> --dest-space <space> clones a workflow, which is how a standard rollout shape is shared between teams that keep their workflows in different Spaces.

Naming a change with a ChangeOrder

Make the change in the base Space first, then create the ChangeOrder. This order matters: a ChangeSet has to exist before the revisions that belong to it, but a ChangeOrder is created after the revisions it names, at the moment you decide to promote them.

# 1. Make the change in the base.
cub function set --space acme-base --unit deployment \
    --change-desc "Release v452: bump the image tag" \
    -- set-image-reference-by-uri ghcr.io/acme/acme ":v452"

# 2. Name it, and say which workflow governs where it goes.
cub changeorder create --space acme-base release-v452 \
    --description "Release v452: bump the image tag" \
    --change-workflow acme-workflows/acme-main-line

Creating a ChangeOrder fixes its scope, per Unit of its Space:

• Where it is headed is recorded as its InScopeSpaceIDs, which the workflow's Stages then divide up. --in-scope-space lists the Spaces, and --component, --where-space-field and --space-filter select them, as described below. Leave all of them out, and wherever its Links reach is where it is headed, which for a component's base is its variants. --change-workflow does not change where it is headed.
• Where each interval starts is the revision the downstream Spaces have already taken — read from the merge cursor of the Links pointing at each Unit. If they disagree, that is an error naming the Unit and the values, rather than a different range being silently promoted to each target.
• Where it ends is each Unit's head revision, or the revision a supplied --end-tag marks on it.
• It mints three Tags of its own, <slug>-co-start, <slug>-co-end, and <slug>-co-publish, and stamps the revisions in the interval with the ChangeOrder. The suffixes differ from a ChangeSet's -start/-end so that a ChangeOrder can carry the same name as the ChangeSet whose boundary it adopts. The third is the release Tag a Release of the change is published at (below).

A ChangeSet is optional. For a single function invocation it adds nothing. Where a change is several edits that must be bounded and locked, open a ChangeSet as described in making changes, close it, and pass its end Tag:

cub changeorder create --space acme-base release-v452 \
    --description "Release v452" \
    --change-workflow acme-workflows/acme-main-line \
    --end-tag acme-base/release-v452-end

Close the ChangeSet first: a ChangeOrder whose end falls inside a ChangeSet that is still open is refused, since that ChangeSet's change is not finished. The supplied Tag is read once, to find each Unit's end revision, and never written to. The ChangeOrder still marks with its own, because it marks Units the ChangeSet never touched.

The Spaces in scope can be selected rather than listed. --where-space-field takes a where expression over Spaces, and --space-filter a saved Filter over Spaces; set both and they are ANDed. --component acme adds ComponentID = '<acme's ID>' to --where-space-field, selecting the Spaces of that component. The server evaluates the selection and records the Spaces it picks, so the ChangeOrder still holds a list, and a Space that matches later is not taken in on its own. cub changeorder update --refresh-spaces evaluates the selection again. A change moves hop by hop along Links, so every Space between the base and a Space in scope must be in scope too, and a selection that skips one is refused:

cub filter create --space platform acme-spaces Space --where-field "Component.Slug = 'acme'"
cub changeorder create --space acme-base release-v452 \
    --description "Release v452, as far as staging" \
    --space-filter platform/acme-spaces --where-space-field "Labels.Stage IN ('Development', 'Staging')"

cub changeorder update --space acme-base release-v452 --refresh-spaces

Every Unit is marked, the way a git commit's tree names a version of every file. The end Tag marks a revision of every Unit of each Space the change is in, other than ConfigHub/YAML Units, and the start Tag each Unit's prior state; the Units the change changed are the ones where the two differ. A Unit the change does not change has both Tags on one revision: where the Spaces that follow it have it, so promoting the change moves nothing it did not change; where it is released, when that is later; or at its head, if nothing follows it and it has never been released. --prior-revisions All on cub changeorder create marks every such Unit at its head instead, so promoting the change brings every Unit up to the base, as a promotion without a ChangeOrder does. It is also the default --prior-revisions for the ChangeOrder's promotions.

cub changeorder get reports Skipped Units — the Units of its own Space it carries no revisions of, and why of each — so a ChangeOrder narrower than you expected says so instead of being silently narrow.

The Links a ChangeOrder follows are UpgradeUnit Links, the clone lineage, by default. --update-type MergeUnits follows MergeUnits Links instead, for downstream Units that merge from the base without having been cloned from it. Promotion then plans each Space from its MergeUnits Links rather than from its Units' upstream, so the Spaces need not be variants, and a Space is promoted after any selected Space its Links merge from. A MergeUnits ChangeOrder changes the Units that are merged from; a Unit of the base that nothing follows by a MergeUnits Link is skipped rather than cloned, since a clone would be followed by an UpgradeUnit Link instead.

In a release pipeline, this is two commands after the build: set the new image reference in the base, then create the ChangeOrder that names the release. Check first that a ChangeOrder of that name does not already exist — creating one is what fixes the range, so a second run of the same version should stop rather than bump the image and then fail on the slug:

if cub changeorder get --space acme-base "release-$VERSION" >/dev/null 2>&1 ; then
    echo "$VERSION has already been released" >&2
    exit 1
fi

Do not use --allow-exists here. It returns the ChangeOrder already there, whose end Tag is on the head as it was then; if the unit has moved since, a later promotion carrying that ChangeOrder walks a range that stops short of this release.

Routine rollouts by invocation

The ChangeOrders above name a change that has already been made in the base, and promotion merges it into each downstream Space along its Links. Many routine changes are not like that: bumping an image tag, setting a label, or rotating a setting is the same operation in every variant, with nothing in a base to merge from. For those, a ChangeOrder can name an Invocation to run in each Space instead, with --update-type Invoke.

Save the operation as an Invocation. Declaring the tag as a parameter makes the Invocation reusable, and each ChangeOrder supplies the value for its rollout:

cub invocation create --space platform set-image Kubernetes/YAML \
    --parameter tag:string:true \
    -- set-image-reference-by-uri \
    --repository-uri=ghcr.io/acme/acme \
    --image-reference='template:{{ .Params.tag }}'

cub changeorder create --space acme-base acme-v453 \
    --description "acme v453" \
    --update-type Invoke --invocation platform/set-image --param tag=:v453 \
    --where-unit "Labels.tier = 'frontend'" \
    --change-workflow acme-workflows/acme-main-line

The Invocation, its parameter values, and the --where-unit or --unit-filter selection of Units are fixed when the ChangeOrder is created, so every Space receives the same change, however far apart in time they are promoted. The ChangeOrder is created before the change is made anywhere, and a Space that has not been promoted has not had it.

From there it moves like any other ChangeOrder. cub variant promote --change-order runs the Invocation in the Spaces of the next Stage, the same prerequisites gate each hop, cub release publish --revision ChangeOrder:<slug> releases it, and cub variant demote takes it back out. The Spaces need not be variants of one another at all, since nothing is merged between them. Two differences follow from there being no upstream:

• Nothing is cloned. A Space that lacks a Unit the selection would cover does not get one.
• The Unit selection is asked again on every read. A Unit added to a Space after the change was made there has not had the Invocation run on it, so the Space reads as not having taken the change until it is promoted again.

cub unit update --upgrade --change-order refuses an Invoke ChangeOrder, since there is no upstream revision to merge.

Rollouts from a source outside the component

Some changes start in a Unit that is not part of the component at all. registrybot, for example, keeps one Unit per container repository, in a Space of its own, holding the tag each named stream currently points at. Updating container images automatically walks through setting it up and rolling out each new image this way.

configHub:
  configSchema: registrybot.confighub.com/v1alpha4
  configName: acme
repository: ghcr.io/acme/acme
streams:
  stable:
    tag: v453
    digest: sha256:41aa0c3b5e9d7f2a8c6e1b4d0f3a9c7e5b2d8f6a4c1e9b7d3f0a6c8e2b5d9f1a

Each variant's workload follows it with a TransformPaths Link to that Unit, created in the variant's Space:

cat <<'EOF' | cub link create --space acme-dev - deployment acme registry-facts --update-type TransformPaths --from-stdin
Protect: true
UpstreamPaths:
  - Name: tag
    Path: streams.stable.tag
    Resource:
      ResourceType: registrybot.confighub.com/v1alpha4
      ResourceName: acme
DownstreamSetters:
  - Parameters: [tag]
    FunctionInvocation:
      FunctionName: set-container-image-reference
      Arguments:
        - ParameterName: container-name
          Value: acme
        - ParameterName: image-reference
          Value: ":{{.Params.tag}}"
          Evaluator: template
EOF

set-container-image-reference takes the tag or digest with its separator, so the template adds the : the tag field lacks; reading streams.stable.digest with @ pins by digest instead. set-image-reference-by-uri takes repository-uri in place of container-name, for workloads whose container is named differently across variants. Protect keeps upgrades from carrying the image reference a variant's upstream took through its own Link, so each variant takes it only through its own Link, in its own stage.

This is the job the Argo CD Image Updater and Flux image automation do by writing straight into each deployment. Here the new tag is one change, and a ChangeOrder rolls it out through the component's ChangeWorkflow like any other. The Links must not be AutoUpdate, since an automatic Link takes the new tag the moment registrybot writes it, ahead of any gate.

When registrybot records a new tag, name that change with a ChangeOrder of the Links' type, created in the component's base:

cub changeorder create --space acme-base acme-stable-v454 \
    --description "acme stable is v454" \
    --update-type TransformPaths --component acme \
    --change-workflow acme-workflows/acme-main-line

Insert and Upsert work the same way, for Insert and Upsert Links. For all three:

• The ChangeOrder follows the Links of its type from the Spaces in scope to Units in Spaces outside its scope. Those Units are its sources, and it marks each one: the end Tag on the revision it is at now (or the revision --end-tag marks), and the start Tag on the oldest revision any Link in scope has taken. The Spaces in scope are required, with --component, --in-scope-space, --where-space-field or --space-filter, since the Links leave from them and there is nothing else to tell targets from sources: a Space in scope is a target, never a source. A selection is evaluated across the organization, so narrow it to the component, with --component or ComponentID = '<component-id>'.
• Every Space takes the same state of the source. Promotion resolves each Space's Links at the revision the end Tag marks, not at the source's head, so a newer tag registrybot records while the rollout is under way does not reach a later stage; that is a change for another ChangeOrder.
• Creating one is refused when a Link in scope is AutoUpdate, or has already resolved past the revision the ChangeOrder would end at, since resolving it to the end would take it back.
• Nothing is cloned, and the Spaces need not be variants. Each Space resolves its own Links, so none waits for another within a stage. --prior-revisions does not apply and is refused.
• The base does not have the change just by holding the ChangeOrder, since the change was made in the sources. It counts only if it is in scope and its own Links have taken it.
• Demoting restores the Spaces in scope and leaves the sources alone.

--merge-end on a resolve works the same way outside a ChangeOrder: cub unit update --patch --resolve "Link:*" --merge-end 12 resolves Insert, Upsert, and TransformPaths Links against revision 12 of the upstream Unit rather than its head.

Promoting stage to stage

cub variant promote moves the change. With --change-order and no Space, it advances the change into the first stage it has not reached:

cub variant promote --change-order acme-base/release-v452

To name the stage explicitly:

cub variant promote --change-order acme-base/release-v452 --target-stage staging

Preview first. --dry-run reports the Units that would be upgraded and added without changing anything, and fails for any Unit the promotion would fail on; -o mutations shows the values that would change:

cub variant promote --change-order acme-base/release-v452 --target-stage staging --dry-run -o mutations

To promote one Space rather than a whole stage — repairing a variant that failed inside a partial stage, or a ChangeOrder with no workflow at all — name it, or select Spaces with --where-space:

cub variant promote acme-staging-euw1 --change-order acme-base/release-v452

What promotion does, per target Space:

1. Clones the Units the change carries that the target does not have yet, at the revision the ChangeOrder starts from — before upgrading, which is the reverse of a plain promote. The change then replays into the clone, so it ends up carrying the revisions the change made everywhere else rather than arriving whole with the change already folded in. A Unit created upstream after the ChangeOrder was fixed is outside the change: it is reported rather than cloned, and a promotion without --change-order is what adds it. A Unit of the Space above that the change does not change, such as one that Space created for itself, is not cloned either, unless --prior-revisions is All.
2. Upgrades the Units that are behind, with the ChangeOrder supplying the range so the merge stops where the change ends rather than at the upstream's head. A change made upstream after the ChangeOrder was created is left behind.
3. Marks what it did. The ChangeOrder's start Tag goes on the target's head before the merge and its end Tag on the revision the merge arrives at, whether the range landed as one revision or as one per source revision. A Unit that already had the change, because it was made there first or set by hand, makes no revision and gets both Tags on its head; the promotion reports it as already having the change.
4. Marks every other Unit of the Space as one the change does not change, and places the release Tag, once the Space's writes are done. That includes the Units the variant created for itself rather than cloning from the base. A variant downstream neither takes the variant's own edits to them nor clones the ones it does not have, unless --prior-revisions is All.

Revisions before the ChangeOrder's start. When a ChangeOrder is promoted into a Space, its start Tag goes on each Unit's head there. Anything that Space wrote into a Unit before then is not part of the change, and the variants downstream of it have not merged it yet. A TransformPaths Link that hashes a ConfigMap into a Deployment is the usual source: changing the ConfigMap writes a new Deployment revision in each Space. --prior-revisions says what the next promotion does with those revisions:

• Include, the default, merges them first, as revisions that do not carry the ChangeOrder, and then promotes the change. The start Tag goes on the revision they leave, so demoting the ChangeOrder later keeps them. This is the same as upgrading the Unit to the ChangeOrder's start and then promoting.
• Skip promotes only the change, as though the Unit had already merged as far as its start. What those revisions changed does not reach the Unit, then or later, since its Link moves past them. A range can still be taken explicitly with cub unit update --merge-source --merge-base --merge-end.
• Error refuses, naming the Unit, its Link, and each revision with where it came from.
• All merges them as Include does, and also marks every Unit of the Space the change does not change at its head, so the next promotion brings them up to this Space too.

Without --prior-revisions, a promotion takes the ChangeOrder's.

cub variant promote --change-order acme-base/release-v452 --target-stage staging --prior-revisions Error --dry-run

A Unit that has already merged past the ChangeOrder's start, such as one upgraded by hand to the upstream's head, is refused whatever --prior-revisions says, since it already has part of what comes after the start. The error lists the revisions it has taken.

Promotion moves configuration only. It does not publish a Release.

The gates are the stage's, not any one Space's. They are evaluated once over the whole membership of the stage before. Naming a later stage does not skip what precedes it: --target-stage prod while staging is unsatisfied is refused, naming each prerequisite and the Space that blocks it:

Failed: unable to promote to stage 'prod', Variant 'staging' has taken change order 'release-v452' but has not released it; unable to promote to stage 'prod', Variant 'staging': requires two-approvers: 2 Approval attestation(s) from eligible attesters who did not write the change; deployment revision 3 has 0 of 2

A stage can land partway. Its Spaces are promoted one at a time; one that fails is reported and the ones after it are still promoted, and the result says how many landed. Running the promotion again is what repairs it: a Space that already took the change is passed over, because a ChangeOrder reaches a Unit once and only once — a Unit already carrying the end Tag has taken the change, whatever has happened to it since.

A promotion that does not complete is recorded on the ChangeOrder, in PromotionFailures: who ran it, when, and each Space it failed or was blocked in, with the error for the Space and for each Unit whose write failed. cub changeorder get shows the most recent one:

cub changeorder get --space acme-base release-v452

-o json lists them all. A dry run records nothing, and neither does a promotion refused by its gates. A failed cub unit update --upgrade --change-order is not recorded either; the command reports the error directly.

The server enforces the gates, for cub variant promote, the UI, and any other caller of the promote API alike. When a gate has to be overridden, such as to ship a fix during an incident, --force promotes past gates that do not hold. It requires --force-reason, and the ChangeOrder records who forced it, when, into which Stage, why, and which gates failed, in its PromotionOverrides:

cub variant promote --change-order acme-base/release-v452 --target-stage prod \
    --force --force-reason "incident 1234"

A promotion takes the options of a cub unit update, --clearance, --guard and --subgroup, and applies them to every write it makes, as a promotion without --change-order does; see promoting a variant. Two differ with a ChangeOrder. --tag is refused, since the ChangeOrder's own Tags mark what it promotes. --protect is accepted only for an Invoke ChangeOrder and for one that follows Links out of the component, which write values rather than merge them; promoting a ChangeOrder along the clone lineage or over MergeUnits Links is a merge, which protection holds paths against.

Upgrading a Unit directly with cub unit update --upgrade --change-order is not a promotion and is not gated: it is how a single Unit is repaired. It clones no missing Units, copies no Links, and marks only the Units it writes, but it does move the change into that Unit, and it takes --prior-revisions too.

Links are copied independently of the change. A promotion into a variant gives it a copy of each Link of the Space above whose source Unit it has, whatever --prior-revisions says, including a Link the Space above added between Units the variant already has. Links are not yet part of what a ChangeOrder marks.

Review and approval

An approval is recorded as an Attestation: an immutable record that a user approved, or rejected, specific revisions in one Space. What a change needs before it may move is declared on the ChangeWorkflow, as attestation prerequisites, and is checked when a promotion or a publish is attempted. Recording an approval does not re-run anything; the next attempt reads it.

Recording approvals

cub variant approve records approvals for one or more Spaces. With a ChangeOrder, it approves the revisions the change arrived at, in its Spaces, or in one Stage of its workflow:

# Approve the change as it stands in every staging Space
cub variant approve --change-order acme-base/release-v452 --stage staging

# Approve it in one Space
cub variant approve acme-staging-euw1 --change-order acme-base/release-v452

# Preview what would be approved
cub variant approve --change-order acme-base/release-v452 --stage prod --dry-run

Without a ChangeOrder, it approves each Unit's head revision in the Spaces named or selected. By default that is the Units with a Target, which is what a Release of the Space would publish. A base has no Target, so pass --all to approve one. --where narrows the Units, and --revision approves a revision other than the head — a number, LastReleasedRevisionNum, Tag:<tag>, ChangeSet:<changeset>, or ChangeOrder:<changeorder>:

cub variant approve acme-dev
cub variant approve acme-base --all
cub variant approve acme-prod --where "Slug LIKE 'deployment-%'"
cub variant approve --where-space "Labels.Stage = 'Development'" --revision ChangeSet:acme-home/release-v452

A Unit with no revision matching the selection, such as one the ChangeOrder never reached, is reported and skipped rather than approved at some other revision. Approving one Unit is --where "Slug = '<unit>'".

--note records the reason, --claim key=value anything else worth keeping with the approval, such as a change ticket, and --expires-in how long it counts for. --reject records a rejection instead, which blocks a requirement until it is revoked:

cub variant approve acme-staging-euw1 --change-order acme-base/release-v452 \
    --reject --note "breaks the EU ingress"

Attestations cannot be edited or deleted. Withdrawing one records a revocation beside it. Find its ID with cub attestation list, then revoke it, and record a new one if you have changed your mind:

cub attestation list --space acme-staging-euw1
cub attestation revoke <attestation-id> --note "looked again"

An approval covers content. It covers the revisions it names, and any later revision of the same Unit with identical content, so an edit that changes nothing does not need approving again. Any revision with different content is unapproved until someone approves it.

Approval is one type of Attestation. cub attestation create --type records others, such as a security review or a reference to a change record in another system, which a workflow can require the same way:

cub attestation create --space acme-prod-use1 --type ChangeRecord \
    --change-order acme-base/release-v452 --claim servicenow.com/change=CHG0012345

Requiring approvals

Declare the requirements under AttestationPrerequisites, and name them in the Stages that need them:

Stages:
  - Name: dev
    WhereSpace: "Labels.Stage = 'Development'"
  - Name: staging
    WhereSpace: "Labels.Stage = 'Staging'"
    Prerequisites: [Validated, Released]
  - Name: prod
    WhereSpace: "Labels.Stage = 'Production'"
    Prerequisites: [Released, Healthy, two-approvers]
    ReleasePrerequisites: [release-manager]
AttestationPrerequisites:
  - Name: two-approvers
    Count: 2
    Description: two approvals of the change in staging, by people who did not write it
  - Name: release-manager
    FromUserIDs: [9b2c0f3e-5d7a-4a61-8f0e-2b6d1c4e7a90, 4f0d2a71-2c8e-4b1e-9a57-6e3c8d9b1f02]
    MaxAge: 72h

Where a requirement is named decides what it reads.

• In a Stage's Prerequisites, it is an entry gate, evaluated in every Space of the Stage before over the Units the change changed, at what a Release of the change would bundle there. "prod requires two-approvers" means two approvals of the change as it stands in staging, recorded with cub variant approve --change-order ... --stage staging.
• In a Stage's ReleasePrerequisites, it gates each Release for the ChangeOrder in the Stage's own Spaces, and is evaluated over the revisions the Release bundles: the ones its release Tag marks. That is how a per-cluster sign-off is required. Release prerequisites may name only attestation prerequisites. cub variant approve --change-order approves those same revisions, so one approval satisfies the entry gate and the release gate alike, including a revision resolve made after the promotion.
• In Final.Prerequisites, it has to hold in the last Stage for the rollout to read as completed.

A requirement holds when every revision it reads has at least Count distinct eligible users with an approval, and no unrevoked rejection from an eligible user. FromUserIDs limits who is eligible, and MaxAge how old an approval may be. Empty FromUserIDs means anyone with the Approve permission on the Space.

The authors of a change do not count. Anyone who wrote a revision of the change in that Space, from the ChangeOrder's start through the revision being approved, is excluded, since separation of duties is the point of requiring approval. A promotion's revisions are written by whoever promoted, so the person who promotes a change into a Space cannot approve it there. AllowAuthors: true turns the exclusion off, which suits a team of one or a demo. IgnoreFail: true counts only approvals and ignores rejections. The ChangeWorkflow entity page describes every field.

A refusal says what is missing, and where:

Failed: unable to promote to stage 'prod', Variant 'staging-euw1': requires two-approvers: 2 Approval attestation(s) from eligible attesters who did not write the change; deployment revision 3 has 1 of 2
Failed: HTTP 422: unable to publish a release of change order 'release-v452' in stage 'prod': requires release-manager: 1 Approval attestation(s) from eligible attesters who did not write the change; deployment revision 3 has 0 of 1

A workflow can require approval of the change in the base before it goes anywhere: name the requirement in the first Stage's Prerequisites, which are evaluated over the base, and approve in the base.

Releasing each stage

Publishing is a separate operation, never a step of promote. A stage that has been promoted but not released is a legitimate resting state — and the Released prerequisite is what makes the next stage wait for it.

Publish per Space, naming the ChangeOrder:

cub release publish --revision ChangeOrder:acme-base/release-v452 acme-staging-euw1

--revision ChangeOrder:<slug> publishes at the ChangeOrder's release Tag, which marks every releasable Unit of the Space: the revision the change arrived at, and every Unit the change did not change as it is deployed. A Release describes the change and not whatever else has happened since, and --dry-run shows what it would bundle before anything is published.

The release Tag follows what finishes the change after the promotion. When a Link within the Space resolves on its own, such as an AppConfig Unit the change updated being rendered into a ConfigMap, the ConfigMap's new revision takes the release Tag; the end Tag stays where the change landed. A fix made in review is adopted by moving the release Tag to it:

cub unit tag ChangeOrder:acme-base/release-v452 --move --revision HeadRevisionNum --space acme-staging-euw1 --unit deployment

It cannot move before where the change landed or what is released, nor to a revision with validation errors, and once a Release in the Space has been published at it, its marks there are fixed.

--revision ChangeOrder:<slug> also makes the Release one for the ChangeOrder, so the Stage's ReleasePrerequisites apply to it, such as an approval of the revisions being released in that Space; see review and approval. Before:ChangeOrder:<slug> names the state before it, which is how the previous state is re-released to roll the change back.

cub release publish takes one Space at a time; releasing a whole stage is that command across the stage's Spaces. See Publishing Releases.

Where has it got to

Where a ChangeOrder has got to is derived when it is read, from the Tags its promotions left and the Releases that have been published, which is why it cannot go stale. The one exception is Stage, which the server records as the change moves: it is advanced when the change is promoted or released, when the ChangeOrder is updated, and when the live status of a Release published for it is written back. It is the latest Stage every Space of which has taken the change, or Completed once the last Stage also satisfies Final, and it never moves backwards. The gates themselves read current state, so a Stage that has degraded since it was promoted still holds up the next one.

cub changeorder get --space acme-base release-v452

reports:

Field	What it says
State	New, InProgress, Resolved, Released — or Aborted, Restored, RestoreReleased
Stage	the latest stage of the governing workflow the change has reached, or Completed
Completed	whether the last stage has satisfied Final.Prerequisites
Where Space	the selection its in-scope Spaces were evaluated from, when it has one
In-Scope Spaces	where it is headed
Resolved Spaces	where it has been fully promoted
Released Spaces	where a Release carrying it has been published
Skipped Units	the Units it carries no revisions of, and why: those of its own Space, or for an Insert, Upsert, or TransformPaths ChangeOrder, its sources

State and Completed are different readings and can legitimately disagree. State never consults live status or approvals, so a workflow whose Final.Prerequisites name Healthy is not complete at Released; and State reduces over every Space in scope, so a last stage that is released and healthy is complete while an earlier stage that was never gated on Released leaves State at Resolved.

All of the Space sets are queryable, so fleet questions are one command:

# Which rollouts are still in flight?
cub changeorder list --where "State = 'InProgress'"

# Has this change reached that Space?
cub changeorder list --space acme-base --where "ResolvedSpaceIDs ? '<space-id>'"

# Which have got past their own base?
cub changeorder list --space acme-base --where "LEN(ResolvedSpaceIDs) > 1"

# Which are in production, but not done?
cub changeorder list --where "Stage = 'prod'"

Revisions carry the ChangeOrder too, upstream and downstream alike, so "what did this change actually do here?" is a revision query in any Space it reached.

Undoing a rollout: abort and demote

Aborting a ChangeOrder is the record that the change is not coming to the Spaces still waiting for it. Set a reason:

cub changeorder update --space acme-base release-v452 \
    --aborted-reason "superseded by v453"

Aborting changes nothing about the Spaces that already took the change — which is exactly the state that needs undoing. cub variant demote is what takes it back out, one Space at a time:

cub variant demote acme-prod-use1 --change-order acme-base/release-v452 \
    --change-desc "back out v452"

Each Unit the ChangeOrder marked there is restored to the revision it was at before the change — one step, however many revisions the promotion made, because the promotion's start Tag marks that state by name.

Preview it first. A Unit whose head has moved past where the change ended has later changes the restore will drop, and --dry-run names them before anything is written:

cub variant demote acme-prod-use1 --change-order acme-base/release-v452 --dry-run

Things worth knowing:

• Demote requires the abort. Undoing a ChangeOrder nobody has said is not coming is a race with whoever is still promoting it.
• Which Units are restored is the ChangeOrder's answer, not a --where of yours: the Units of that Space its start Tag marks. A Unit it never marked is an error rather than a Unit passed over.
• It is idempotent, like promotion. A Unit already carrying the restore Tag is left alone, so a demote that landed partway is run again to reach the Units it did not — and a Unit edited forward since it was undone keeps that work.
• A ChangeOrder travels one way. An aborted one is refused by every promotion, and its reason cannot be cleared once anything has been restored. A change to promote after an undoing is a new ChangeOrder.
• The ChangeOrder's own Tags stay where they are. A Space that took the change took it, whatever has happened since. The restore Tag is what says it was taken back out again, and State reconciles the two: Restored once every Space that took the change has been demoted, RestoreReleased once every Space that had released it has released the restored revisions.
• Nothing is promoted or released for you. Each Space is demoted on its own account, and publishing the restored revisions is a separate cub release publish — because what a Space goes back to has to be released where the change was released. Re-applying anything the restore dropped is a forward change to make afterwards.

To undo a change that was never named by a ChangeOrder, or to take out one range of revisions inside a single Space, revert it instead.

Related reading

• Making Changes — making, grouping, and undoing a change within one variant
• Creating and managing variants — clone, upgrade, protection, conflicts
• Managing environments — the Space labels stages select on
• Publishing Releases — the Release command reference, and what blocks a publish
• Attestation and ChangeWorkflow — the entities behind approvals and their requirements
• Advanced merging techniques — what a promotion's merge does