Updating container images automatically

When a new image is pushed to a container registry, something has to notice it and change every deployment that should run it. The Argo CD Image Updater and Flux image automation do that by watching the registry and writing the new tag straight into each application's manifests or Git repository, and Renovate and Dependabot do it by opening a pull request per repository. Each deployment is updated on its own, and the order the environments get the new image in is whatever order the writes happen in.

In ConfigHub the same job is split into three parts, each of which uses a mechanism ConfigHub already has:

• registrybot records what the registry holds. It watches container repositories and keeps one Unit per repository, a fact Unit, holding the tag each named stream currently points at, such as the highest release version. It writes nothing else.
• Each variant's workload follows a fact Unit through a TransformPaths Link. The Link reads the stream's tag from the fact Unit and sets the image reference in the variant's Deployment. Which stream a workload follows is decided by the Link, so "stable" is defined once, next to the repository, rather than in every consumer.
• A ChangeOrder of type TransformPaths rolls out the new tag. It names the revision of the fact Units that holds the new tag, and promoting it through a multi-stage ChangeWorkflow resolves each stage's Links at that revision, with the workflow's gates, such as approvals and a healthy release in the stage before, between the stages.

The examples on this page use a component shop with two images, ghcr.io/acme/shop (run by the deployment Unit) and ghcr.io/acme/shop-ui (run by the ui Unit), a base Space shop-base, a staging variant shop-staging labeled Stage=test, and a production variant shop-prod labeled Stage=prod. See managing environments for the Stage label.

Running registrybot

registrybot is a small process that authenticates to ConfigHub as a worker identity and reads the GitHub Packages API. It supports ghcr.io. Its README is the reference for its configuration and deployment; this section covers what you need to set it up for image updates.

Fact Units

For every repository it watches, registrybot maintains one fact Unit of toolchain AppConfig/YAML, named <owner>-<name> by default and labeled registrybot.confighub.com/repository=<repository>. For ghcr.io/acme/shop, the Unit acme-shop holds:

# Facts about ghcr.io/acme/shop, observed by registrybot from the GitHub Packages API.
# Do not edit: the next observation overwrites this unit. Link to it instead.
configHub:
  configSchema: registrybot.confighub.com/v1alpha4
  configName: shop
repository: ghcr.io/acme/shop
registry: ghcr.io
owner: acme
source: github-packages
url: https://github.com/orgs/acme/packages/container/package/shop
streams:
  main: null
  stable:
    tag: v1.4.0
    digest: sha256:31dbc1f5858dc74193883e110fcbfe90116cedc77dd87d56e5d006557d84b148

Each stream is the one tag it selects, with that tag's digest, or null when no tag matches. There is deliberately no list of tags: a Link cannot name an entry of a list by key, and the Unit's revisions already record what each stream pointed at, and when. registrybot writes a new revision only when a stream changes, with a change description such as registrybot observed ghcr.io/acme/shop: main=none stable=v1.4.0@31dbc1f5858d, so the revision history of a fact Unit is the history of its repository's releases.

The AppConfig toolchains read configHub.configSchema and configHub.configName as the resource type and name, so a Link names the fact as resource type registrybot.confighub.com/v1alpha4 and resource name shop. Resource types are compared without their version segment, so a Link keeps working when registrybot's schema version changes.

Do not edit a fact Unit. registrybot is its only author, and its next observation overwrites any edit.

The configuration document

What to watch is itself configuration, in an AppConfig/YAML Unit that registrybot re-reads every poll cycle, so editing that Unit reconfigures the running bot. Keep it and the fact Units in a Space of their own, outside every component, since the facts are not part of any one component's configuration and several components may follow the same repository:

pollInterval: 5m
defaults:
  # Tags that are not images anyone deploys: per-commit and per-PR builds, build caches,
  # single-architecture images, and cosign signatures and attestations
  exclude: ["^sha-", "^pr-", "^buildcache-", "^sha256-", "^(amd64|arm64)-"]
  streams:
    stable: { semver: ">=0.0.0" }
    main: { pattern: "^main$" }
repositories:
  - repository: ghcr.io/acme/shop
  - repository: ghcr.io/acme/shop-ui

A stream with semver selects the highest tag that parses as a release version (v1.2.3 or 1.2.3) and satisfies the constraint; prerelease versions are included only when the constraint admits them, as >=2.0.0-0 does. A stream with only pattern selects the most recently pushed tag matching it. The built-in stable stream, the highest release version with prereleases excluded, is always present unless you redefine it. Streams can also be defined per repository:

repositories:
  - repository: ghcr.io/acme/shop
    streams:
      stable: { semver: ">=1.0.0 <2.0.0" }
      next:   { semver: ">=2.0.0-0" }

With discovery.fromWebhooks: true, a GitHub package webhook event for a repository the document does not list creates a fact Unit for it, from defaults, provided its owner is in discovery.owners and it matches no discovery.exclude pattern. With an organization-level webhook, every image the organization publishes then gets a fact Unit without anyone editing the document. To stop watching a discovered repository, delete its fact Unit.

Deploying it

Create the Space, the worker identity registrybot authenticates as, and the configuration Unit:

cub space create registry-facts
cub worker create registrybot --space registry-facts
cub worker get-envs registrybot --space registry-facts
cub unit create --space registry-facts --toolchain AppConfig/YAML registrybot registrybot-config.yaml

The worker's identity inherits the Organization role of the user who created it, which is what lets it create and write Units. Then run the bot with the environment variables cub worker get-envs prints, CONFIGHUB_URL, REGISTRYBOT_CONFIG_SPACE=registry-facts, and a GITHUB_TOKEN with read:packages. Fact Units go in the configuration Unit's Space unless defaults.space says otherwise. registrybot publishes its own Kubernetes configuration as an OCI bundle, so it can be deployed from ConfigHub like any other component:

cub variant upload --component registrybot --variant base oci://ghcr.io/confighub/configs/registrybot

The Deployment reads its credentials from a Secret named registrybot-secrets, which the bundle deliberately does not contain; create it out of band, with the keys CONFIGHUB_WORKER_ID, CONFIGHUB_WORKER_SECRET, GITHUB_TOKEN, and optionally GITHUB_WEBHOOK_SECRET. Run a single replica.

Polling alone is complete. A webhook makes registrybot look sooner: set GITHUB_WEBHOOK_SECRET, expose POST /webhooks/github to GitHub, and add a webhook on the GitHub organization or repository with content type application/json, the same secret, and only the Packages event. One push of a multi-architecture image arrives as several events over a few minutes, so registrybot waits until deliveries for a repository stop for webhooks.settle (2 minutes by default) and then records the result once. Nothing in a webhook payload is used as data; it only decides when to look.

Within one poll cycle each repository has its fact Unit:

cub unit list --space registry-facts

Following a fact Unit

Leave a placeholder in the base

The variants set their own image, so the base does not need a real one. Give it a placeholder image tag instead:

cub function set --space shop-base --unit deployment \
    --change-desc "the image tag is set by each variant's Link" \
    -- set-image-reference-by-uri ghcr.io/acme/shop ":confighubplaceholder"

With a vet-placeholders trigger on the Units with a Target, a variant whose Link has not set its image yet has a validation error, and cannot be released by mistake with the placeholder in it.

A TransformPaths Link in each variant

Create a Link from each variant's workload to the fact Unit, in the variant's Space, and resolve it once to set the current image:

for space in shop-staging shop-prod ; do
    cub link create --space "$space" shop-image deployment acme-shop registry-facts \
        --update-type TransformPaths --protect \
        --upstream-path 'tag=registrybot.confighub.com/v1alpha4:shop:streams.stable.tag' \
        --downstream-setter 'set-image-reference-by-uri ghcr.io/acme/shop template::{{.Params.tag}}'
    cub link create --space "$space" shop-ui-image ui acme-shop-ui registry-facts \
        --update-type TransformPaths --protect \
        --upstream-path 'tag=registrybot.confighub.com/v1alpha4:shop-ui:streams.stable.tag' \
        --downstream-setter 'set-image-reference-by-uri ghcr.io/acme/shop-ui template::{{.Params.tag}}'
    cub unit update --space "$space" --patch --where "Slug IN ('deployment', 'ui')" \
        --resolve "Link:UpdateType = 'TransformPaths'" --change-desc "follow the stable stream"
done

Create the Links in every variant you add later, too. Each part of the Link has a reason:

• --upstream-path reads the stream's tag, as <name>=<resource-type>:<resource-name>:<path>. Read streams.stable.digest instead to pin by digest, and write template:@{{.Params.digest}} in the setter: the setter takes the tag or digest with its separator, and tag and digest do not include one.
• set-image-reference-by-uri sets the image of every container, init containers included, whose image is from that repository, whatever the container is named. A workload that runs the same image in a migration init container gets the new tag in both. To set one container only, use set-container-image-reference <container-name> instead.
• --protect records the image the Link writes as the variant's own, so an upgrade from the base never carries the base's image over it. Without it, a change to the image in the base, or the placeholder, would be merged into each variant by its next upgrade. To also stop other writers, guard the field and give the Link a clearance for it; see guarding values against every change.
• No --auto-update. An automatic Link takes the new tag the moment registrybot writes it, ahead of any gate, and a TransformPaths ChangeOrder refuses a Space with one. A Link without it reports itself as stale in cub link list when the fact Unit moves past the revision it last took, and waits for a resolve.

Variants that should run every new build as soon as it is pushed, such as a development environment following the main stream, can use an --auto-update Link instead, and are then left out of the ChangeOrder's scope. main is a moving tag, so follow it by digest.

Rolling out a new tag

A multi-stage ChangeWorkflow

The workflow says which Spaces get the new tag first, and what must hold before the next stage gets it. Keep it in a Space of its own:

cub space create shop-workflows
cub changeworkflow create --space shop-workflows shop-images \
    --stage test --stage prod \
    --stage-prerequisites 'prod=Released;Healthy;prod-approval' \
    --stage-release-prerequisites 'prod=prod-approval' \
    --final-prerequisites 'Released;Healthy' \
    --attestation-prerequisite prod-approval \
    --attestation-prerequisite-description 'prod-approval=approval by a release manager before promotion to and release of prod' \
    --attestation-prerequisite-from-user-ids 'prod-approval=9b2c0f3e-5d7a-4a61-8f0e-2b6d1c4e7a90;4f0d2a71-2c8e-4b1e-9a57-6e3c8d9b1f02' \
    --attestation-prerequisite-allow-authors prod-approval=true

Each --stage selects the Spaces whose Stage label is its name. The workflow can also be written in a file, in the fields cub changeworkflow get -o json prints, and passed with --filename; see declaring a ChangeWorkflow.

The test stage has no prerequisites. A TransformPaths ChangeOrder's change is in no Space until it is promoted into one, so it refuses a workflow whose first stage has prerequisites. The prod stage can be entered only once every test Space has released the change, the release is reported healthy, and a release manager has approved it, and a Release of it in prod needs that approval too. --attestation-prerequisite-allow-authors lets a release manager who promoted the change approve it; leave it out to require someone who did not. See prerequisites and review and approval.

Selecting the Spaces

A TransformPaths ChangeOrder has to be told which Spaces it is headed for, since its Links leave from those Spaces and there is nothing else to tell its targets from its sources. Save the selection as a Filter over Spaces, so that every ChangeOrder for the component uses the same one:

cub filter create --space shop-workflows image-stages Space \
    --where-field "Component.Slug = 'shop' AND Labels.Stage IN ('test', 'prod')"

Select only the Spaces that follow the fact Units through Links. The base and any intermediate variants have no such Links, and a Space whose Links are AutoUpdate would be refused.

Creating the ChangeOrder

Once registrybot has recorded the new tag, the Links in scope are stale:

cub unit data --space registry-facts acme-shop
cub link list --space shop-staging

Name the change with a ChangeOrder in the base:

cub changeorder create --space shop-base release-v1.5.0 \
    --description "Release v1.5.0 of shop and shop-ui" \
    --update-type TransformPaths --space-filter shop-workflows/image-stages \
    --change-workflow shop-workflows/shop-images

The ChangeOrder follows the Links of the Spaces in scope to the fact Units, and marks the revision each fact Unit is at now as the end of the change. One ChangeOrder carries both images, since both Links are in scope, so one promotion moves a Space's server and the UI that goes with it together. Create it after both fact Units have the new tag: it ends at the revision each is at when it is created. Every stage then gets the same state of the facts, so a newer tag registrybot records while the rollout is under way does not reach a later stage; that is a change for another ChangeOrder. Rollouts from a source outside the component describes what such a ChangeOrder marks and when creating one is refused.

Promoting, approving, and releasing

From here the ChangeOrder moves like any other. Each promotion advances it one stage, resolving the Links of that stage's Spaces at the revision the ChangeOrder ends at. Preview each one first; -o mutations shows the image references that would change:

cub variant promote --change-order shop-base/release-v1.5.0 --dry-run -o mutations
cub variant promote --change-order shop-base/release-v1.5.0
cub release publish --revision ChangeOrder:shop-base/release-v1.5.0 shop-staging

Once staging is released and healthy, approve the change as it stands there, promote it into production, approve it there, and release it:

cub variant approve --change-order shop-base/release-v1.5.0 --stage test
cub variant promote --change-order shop-base/release-v1.5.0
cub variant approve --change-order shop-base/release-v1.5.0 --stage prod
cub release publish --revision ChangeOrder:shop-base/release-v1.5.0 shop-prod

cub changeorder get --space shop-base release-v1.5.0 shows where the change has got to: its state and stage, the Spaces that have taken it and those that have released it, and the most recent promotion. See promoting stage to stage and releasing each stage.

Rolling back

To take a release back out, abort the ChangeOrder and demote it, then publish each Space it was released in:

cub changeorder update --space shop-base release-v1.5.0 --aborted-reason "v1.5.0 fails health checks"
cub variant demote --change-order shop-base/release-v1.5.0
cub release publish shop-staging
cub release publish shop-prod

Demoting restores the variants' Units to the revisions they were at before the change. The fact Units are outside the ChangeOrder's scope and are left alone: they still say what the registry holds.

Automating the release

The image build is the natural place to create the ChangeOrder, once the images are pushed. A release script waits for registrybot to record the new tag in each fact Unit, then creates the ChangeOrder. It checks first that a ChangeOrder for the version does not already exist, so that a second run stops instead of failing partway. Promotion and release stay separate, later decisions:

#!/usr/bin/env bash
set -euo pipefail

VERSION="$1"
NAME="release-$VERSION"

if cub changeorder get --space shop-base "$NAME" >/dev/null 2>&1 ; then
    echo "$NAME already exists; $VERSION has been released" >&2
    exit 1
fi

for unit in acme-shop acme-shop-ui ; do
    until [[ "$(cub unit data --space registry-facts "$unit" | yq .streams.stable.tag)" == "$VERSION" ]] ; do
        sleep 30
    done
done

cub changeorder create --space shop-base "$NAME" \
    --description "Release $VERSION of shop and shop-ui" \
    --update-type TransformPaths --space-filter shop-workflows/image-stages \
    --change-workflow shop-workflows/shop-images

Do not pass --allow-exists to cub changeorder create here: it returns the ChangeOrder already there, which ends where the fact Units were when it was created. Run the script as a service account that can create ChangeOrders in the base Space.

Other ways to roll out an image

• Without registrybot, a ChangeOrder of type Invoke runs a saved set-image-reference-by-uri Invocation in each Space as it is promoted, with the tag as a parameter of the ChangeOrder. See routine rollouts by invocation.
• Through the base, set the image tag in the base and create an UpgradeUnit ChangeOrder, which promotion merges into each variant along the clone lineage, as in naming a change with a ChangeOrder. Every variant then runs the base's image, and the variants' image fields must not be protected.

With fact Units and TransformPaths Links, the registry rather than the base says which tag is current, each variant states which stream it follows, and the history of what each stream pointed at is kept in the fact Units, apart from the component's own changes.