Skip to content

ChangeWorkflow

A ChangeWorkflow is the declared series of Stages a change moves through on its way from where it is authored to everywhere it must land, with the conditions that must hold before a change may enter each Stage, before it may be released in one, and before the rollout counts as complete. It is the definition: the pattern. A ChangeOrder is the invocation: one named change moving through it.

The order is a partial order: sequential between Stages, parallel within one. A Stage is a wave. Everything in it may be promoted in any order, or all at once, and the waves are ordered. A fleet of a thousand clusters cannot be a sequence of a thousand steps.

A ChangeWorkflow is a child of a Space, like a Filter or a Tag, and is created, listed, and edited with cub changeworkflow. Keep workflows in a Space of their own: a workflow is not part of the component it governs, and a definition in a base Space would be cloned into every variant by cub variant create.

Stages:
  - Name: dev
    WhereSpace: "Labels.Stage = 'dev'"
  - Name: staging
    WhereSpace: "Labels.Stage = 'staging'"
    Prerequisites: [Validated, Released]
  - Name: prod
    WhereSpace: "Labels.Stage = 'prod'"
    Prerequisites: [Released, Healthy, not-frozen, approved]
    ReleasePrerequisites: [release-approved]
Final:
  Prerequisites: [Released, Healthy]
CustomPrerequisites:
  - Name: not-frozen
    Expression: "cel:!('code-freeze' in Space.Annotations) || Space.Annotations['code-freeze'] != 'true'"
    Description: the space is not under a code freeze
AttestationPrerequisites:
  - Name: approved
    Count: 2
    Description: two approvals of the change in staging, by people who did not write it
  - Name: release-approved
    FromUserIDs: [9b2c0f3e-5d7a-4a61-8f0e-2b6d1c4e7a90]
    MaxAge: 72h

Stages

WhereSpace is a where expression over Spaces, so Stage membership is by selector rather than by an enumerated list: a variant added to a Stage later is covered without editing the definition. The standard Space labels, Stage, Environment, and Region, are what Stages usually select on; cub variant create and cub variant upload set Stage with --stage. An empty WhereSpace selects every Space the ChangeOrder is headed for.

A Stage's members are the Spaces its selector picks out of its ChangeOrder's InScopeSpaceIDs. The ChangeOrder says where a rollout goes and the workflow says in what order, so the same definition resolves to different Spaces under different ChangeOrders, which is what lets one workflow be shared by several components unchanged. Nothing else is implied, the component included. A WhereSpace can name a component, as ComponentID = '<component-id>', but that ties the workflow to that component. A ChangeOrder headed for no Space has no Space in any Stage.

The definition does not name where the change starts. The base is the Space its ChangeOrder is created in, so the starting point is stated once, by the invocation.

Prerequisites

A Stage names its gates in two lists, and Final in a third.

Prerequisites are entry gates: what must hold to get into a Stage, evaluated over every Space in the Stage before it when a promotion is attempted. The first Stage's are evaluated over the Space the ChangeOrder was created in, where a change that follows Links already is. An Invoke or fan-out change is in no Space until it is promoted into one, so a ChangeOrder of either kind refuses a workflow that gives its first Stage any. Having taken the change is checked whatever is declared; the prerequisites are checks on top of that. Three are built in:

  • Validated: no Unit the change changed in the Space has Validation Errors on what a Release of the change would bundle there: the Revision its release Tag marks, or the one its end Tag marks for a Unit no Release bundles. It needs no Release, so a Stage with no release Target can still require that the change passed validation there.
  • Released: the Space has published a Release carrying the change. A Space with no release Target has nothing to release, so it passes; it is not counted in the ChangeOrder's ReleasedSpaceIDs.
  • Healthy: the Space has published a Release carrying the change, and the live status of its latest published Release, written back by the GitOps operator, reports Synced and Healthy, with no operation Running or Failed. It implies Released. A Release the operator has not reported on yet is not healthy, whatever the Release before it reported, and a Space with no release Target can never satisfy it.

Prerequisites are reads of current state, never cached from when the change was promoted, so a Stage that has degraded since it was promoted does not open the gate after it.

ReleasePrerequisites gate publishing a Release for the ChangeOrder in one of the Stage's own Spaces, and are evaluated over the Revisions the Release bundles, which are the ones the ChangeOrder's release Tag marks. They may name only attestation prerequisites, since the built-in and custom gates read the state of a Stage after the change has moved through it, which a publish into the Stage cannot answer. A publish is for a ChangeOrder when it names one with cub release publish --revision ChangeOrder:<slug>. A refused publish leaves no Release behind.

Final.Prerequisites are evaluated against the last Stage and say when the rollout is completed, rather than when a hop may happen. No promotion reads them, since there is no hop left to gate once the change is in the last Stage. They are named the same way a Stage's Prerequisites are, and are where "the change is not done until it is healthy in production" or "until someone signs off on production" belongs.

Custom prerequisites

CustomPrerequisites declares gates the built-in ones do not cover. Each has a Name, which is what a Stage or Final lists, a CEL Expression carrying the cel: prefix, and an optional Description of what it checks, which is where the reason lives when a promotion reports the gate by name. The expression is evaluated for each Space of the Stage before, with three variables bound to the JSON the API returns for each entity:

  • Space: the Space being evaluated.
  • ChangeOrder: the ChangeOrder being promoted.
  • Release: the earliest published Release of the Space carrying the change, or null when there is none.

Since the entities' Labels and Annotations are within reach, anything outside ConfigHub that can annotate a Space or a Release can gate a rollout, such as a code freeze flag or the result of a load test. Expressions are compiled when the workflow is written, so one that could never evaluate is refused then, not when it would hold up a promotion. An expression that does not produce a boolean fails the gate rather than reading as false.

Attestation prerequisites

AttestationPrerequisites declares the Attestations, such as approvals, that a Stage's Prerequisites, its ReleasePrerequisites, or Final may require. The fields are:

  • Name: what a Stage or Final lists.
  • Type: the Attestation type that counts. Approval by default.
  • Count: how many distinct attesters must record a Pass. 1 by default.
  • FromUserIDs: the users whose Attestations count. Empty means anyone with the Approve permission on the Space. Naming an integration's service account is how "the change record must be approved in ServiceNow" is expressed.
  • AllowAuthors: by default, an attester who wrote any Revision of the change does not count, since separation of duties is what approval exists to provide. true counts them.
  • MaxAge: how old an Attestation may be and still count, as a duration such as 72h.
  • IgnoreFail: by default, an unrevoked Fail from an eligible attester fails the requirement however many passes there are. true counts only passes.
  • Description: what the requirement is for.

FromGroupIDs and DistinctGroups, for requiring approvers from particular groups, are reserved and refused until Attestations record the groups of their attesters.

A requirement is evaluated over a set of Revisions, and holds when it holds for every one of them. In an entry gate, the set is one Revision of each Unit the change changed in each Space of the Stage before, at what a Release of the change would bundle there, so "prod needs two approvals" means two approvals of the change as it stands in staging, and an approval of what staging releases satisfies it. In a release gate, it is the Revisions the Release bundles. For each Revision, the Attestations of Type that cover it, including those of an earlier Revision of the same Unit with the same content, are collected; revoked, expired, too old, ineligible, and author Attestations are discarded; and the distinct users with a Pass are counted.

The authors of a change are the users who wrote its Revisions in the Space: every Revision after the one the ChangeOrder's start Tag marks, through the Revision evaluated. A promotion's Revisions are written by whoever promoted, so a user who promotes a change into a Space cannot then approve it there. Automated Revisions exclude nobody.

Names

Prerequisite names are unique within a workflow across all three kinds, and a declared name may not shadow a built-in one. A Stage or Final naming a prerequisite that is neither built in nor declared is refused when the workflow is written.

How a ChangeOrder picks one up

cub changeorder create --change-workflow <workflow> names the workflow, and the server copies its definition onto the ChangeOrder. Every promotion and release of that ChangeOrder is judged against the copy, so editing the workflow part way through a rollout cannot change the rules a change already started under. The edit applies to ChangeOrders created afterwards. A change that has to move under the new rules is a new ChangeOrder.

The server enforces the gates for every client: cub variant promote, the UI, and the promote API are refused alike when an entry gate does not hold, and a publish is refused when a release gate does not. --force with a reason promotes past failing entry gates, and is recorded on the ChangeOrder.

As a ChangeOrder moves, the server records the Stage it has reached in its Stage, and Completed once the last Stage satisfies Final.

See promoting changes for the workflow in practice.

How a ChangeSet picks one up

A workflow with no Stages governs a ChangeSet instead. cub changeset create --change-workflow <workflow> names it, the server copies it onto the ChangeSet, and its Final prerequisites gate closing the ChangeSet: they are evaluated over every Space it was opened in, against the latest revision it made of each unit, and a close is refused unless they all hold, or is forced with a reason that is recorded on the ChangeSet. Final can name Validated, custom prerequisites, which read Space and ChangeSet (ChangeOrder and Release are null), and attestation prerequisites. Released and Healthy are refused, because a ChangeSet's change is released by a ChangeOrder after the ChangeSet closes. A ChangeOrder refuses a workflow with no Stages, which would have nowhere to promote it, and a ChangeSet refuses one with Stages. See gating the close with a ChangeWorkflow.

Requiring a workflow

A Component can require that changes to its variants go through a ChangeWorkflow, and limit which ones. --allowed-change-workflow names the workflows it allows, and --change-workflow-required requires one; both need Manage permission on the Component:

cub component update --patch acme --allowed-change-workflow acme-workflows/acme-main-line,acme-workflows/release-review --change-workflow-required

A ChangeOrder or ChangeSet is governed by one workflow, so that workflow has to be one every Component it reaches allows. A ChangeOrder is checked against the Component of its own Space and of every Space it is headed for, when it is created and when Spaces are added to its scope. A ChangeSet is checked against the Component of each Space whose units it is opened on, as they join it. When the Component requires a workflow, a promotion or release without one is refused, and so is opening a ChangeSet that has none. A Component that allows ChangeSets as well as rollouts lists a workflow of each kind.