Make and Undo a Change
It's not uncommon to need to make an urgent operational change to some workload or infrastructure in some environment. For example, it could be to scale up a workload, horizontally or vertically, to handle unexpected load, either by changing the workload itself or by changing autoscaling limits.
With Configuration as Code, such as helm charts, making a change through the normal configuration change process can take a long time due to git mechanics and processes (branching, permissions, reviews, CI, etc.) and can have a large blast radius in the case that the value that needs to be changed isn't already factored into environment-specific values. So often operators "break glass": they suspend GitOps reconciliation, change resources in clusters directly, update the configuration in git later if necessary, and re-enable GitOps reconciliation afterward. In the case that the change was temporary, the GitOps operator can eliminate the drift by reconciling from the state in git that never represented the change. That also means there was no record of the change and the change didn't pass through the regular validation checks.
With ConfigHub, the resources for a single environment can easily be changed and released, retaining the history and validation provided by ConfigHub without concern about a larger blast radius than intended. Fine-grained permissions can authorize changes per environment.
cub function set --space cubbychat-prod --unit backend --protect set-replicas 5 -o mutations --change-desc "Temporary boost in capacity to handle news event"
cub release publish cubbychat-prod
source ~/.confighub/clusters/prod.env
kubectl get pods -n cubbychat
At any step, we can look at the configured resource we changed as easily as using kubectl.
cub k8s get --space cubbychat-prod deployment backend --show detail
Since that change was expected to be temporary, I'd like to be able to revert it. Of course, I could use set-replicas to change it back, but in the case of a larger set of changes I'd like a way to just undo the changes.
ConfigHub automatically stores a new revision every time changes are made to configuration.
cub revision list --space cubbychat-prod backend
Publishing a release attaches a tag to the released revisions. We can use the tag from the previous release to specify the revision we want to restore.
cub unit update --patch --space cubbychat-prod backend --restore Tag:release-2 -o mutations --change-desc "Revert capacity boost"
cub release publish cubbychat-prod
Restore creates a new revision with the previous configuration data and protection settings.
cub revision list --space cubbychat-prod backend
cub unit get --space cubbychat-prod backend -o mutations
That's how ConfigHub supports autosave and undo, and makes it as easy to undo a change as it is to make a change.
Where to go next
- Scale it up. The value of this model compounds with fleet size. The Components view example walks a six-app, seven-target promotion setup.
- Guard the flow. Validation and policies run functions as triggers to catch bad config before it ships.
Finish
Cleanup the cluster and ConfigHub contents