Monitoring with Metrics
ConfigHub's processes record their metrics with OpenTelemetry and serve them in the Prometheus text format, through the OpenTelemetry Prometheus exporter. Any system that can scrape a Prometheus endpoint can collect them, including Prometheus itself and the OpenTelemetry Collector's Prometheus receiver. This page lists what each process exposes and how to collect it with Prometheus or the OpenTelemetry Collector.
The exporter turns each OpenTelemetry instrument name into a Prometheus metric name: dots become underscores, counters get a _total suffix, and a unit adds a suffix such as _seconds or _bytes. The instrument argobot.syncs, for example, is exported as argobot_syncs_total, and its attributes become labels. This page uses the exported names, which are the ones you query.
Three processes serve metrics:
| Process | Where it runs | Endpoint | Default port |
|---|---|---|---|
| ConfigHub server | Your cluster, if you self-host ConfigHub | /internal/metrics |
9090 (the API port, PORT) |
| argobot | Each cluster it syncs, next to Argo CD, Flux, or Sveltos | /metrics |
9090 (METRICS_ADDR) |
| registrybot | Wherever you run it | /metrics |
9090 (METRICS_ADDR) |
On the hosted service at hub.confighub.com the server is operated for you and its metrics are not exposed. argobot and registrybot run in your own infrastructure either way, so you can always scrape them.
None of the endpoints require authentication. Keep them inside your network: don't route them through an ingress.
ConfigHub server
The server serves its metrics at /internal/metrics on the API port, 9090 by default. The OCI registry port, 9092 by default, serves the same endpoint, and both return the same metrics: the API's and the registry's requests are recorded together and told apart by the server label. Scrape only the API port; scraping both doubles every series.
The cub server plugin's ingress does not route /internal, so the endpoint is reachable only from inside the cluster. If you deploy from your own manifests, don't route /internal either. It also serves pprof profiles.
HTTP request metrics
Every request to the API server and to the OCI registry is recorded.
| Metric | Type | Labels |
|---|---|---|
requests_total |
counter | server, http_request_method, http_route, http_response_status_code, server_address, url_scheme |
http_server_request_duration_seconds |
histogram | server, http_request_method, server_address, url_scheme |
http_server_active_requests |
gauge | server, http_request_method, server_address, url_scheme |
http_server_request_body_size_bytes |
histogram | same as requests_total |
http_server_response_body_size_bytes |
histogram | same as requests_total |
The labels are:
server:apifor the API server,ocifor the OCI registry.http_request_method:GET,POST, and so on. Methods outside the standard set are recorded as_OTHER.http_route: the route pattern, such as/api/space/:space_id, not the requested path, so the number of series stays bounded. It is empty when no route matched.http_response_status_code: the status code of the response.server_address: the host the request was sent to, from itsHostheader. Always empty for the OCI registry, which any client can reach with anyHostheader.url_scheme:httpsif the request arrived over TLS or a proxy on a private address said so, otherwisehttp.
Things to know when you query them:
- Request duration has no route or status labels. You can break latency down by server and method, but not by route or status code. Durations are measured in whole milliseconds, and the buckets run from 5 ms to 120 s.
- The size histograms' buckets are 1 KB, 5 KB, 10 KB, 100 KB, 500 KB, 1 MB, 5 MB, 10 MB, 100 MB, and 1 GB.
requests_totalis a generic name. Other targets may export a metric with the same name, so select the server's series by a label your scrape configuration adds, such asnamespace="confighub", or byotel_scope_name="echo", which every HTTP metric of the server carries.
Usage
usage_active_users_total(counter): the number of distinct users who have made an authenticated request since the server started. It is kept in memory, so it goes back to zero when the server restarts, and each replica counts on its own.
Runtime and process
The endpoint also carries the Prometheus Go client's standard collectors: Go runtime metrics (go_goroutines, go_memstats_*, go_gc_duration_seconds, and so on) and process metrics (process_cpu_seconds_total, process_resident_memory_bytes, process_open_fds, and so on). OpenTelemetry adds target_info, one series of which carries service_name="confighub".
argobot
argobot serves its metrics at /metrics on METRICS_ADDR, :9090 by default. Every series carries a controller label naming the GitOps controller argobot drives: argocd, flux, or sveltos.
| Metric | Labels | Counts |
|---|---|---|
argobot_syncs_total |
controller, result |
Sync requests to the GitOps controller, one per Argo CD Application, Sveltos profile, or Flux OCIRepository a Release event resolved to. result is success or failure. |
argobot_events_unresolved_total |
controller |
release.published events that resolved to no Application, profile, or OCIRepository, so nothing was synced. A steady rise usually means a sync target is misnamed or its OCI URL names a different Space. |
argobot_status_reports_total |
controller, result |
Passes of the live status reporter over one watched object. success: the object's Release was found, whether or not anything had to be written. skipped: there was nothing to record on, because the object is gone or names no Space or Release argobot can see. failure: the pass failed and will be retried. |
registrybot
registrybot serves its metrics at /metrics on METRICS_ADDR, :9090 by default. This is a separate port from its webhook receiver (:8080), so exposing the webhook does not expose the metrics.
| Metric | Labels | Counts |
|---|---|---|
registrybot_reconciles_total |
result |
Reconciles of one repository's fact Unit, from a poll or a webhook. written: the fact Unit got a new revision, including when it was created. unchanged: the facts were already current. failure: the reconcile failed and is left to the next poll. A steady failure rate usually means the GitHub token or the service account's grants are wrong; registrybot's /status endpoint names the repository and the error. |
registrybot_discoveries_total |
result |
Repositories a signed webhook named that registrybot was not watching. admitted: the discovery policy let the repository in, so it now has a fact Unit. refused: it did not, because discovery is off or the owner or an exclude pattern rules it out. |
What argobot and registrybot have in common
- Every series exists from the start, at zero. A
failureseries is there before the first failure, sorate()andincrease()see the first increment and an alert on it has data to evaluate from the moment the process starts. - Counters restart from zero whenever the process does. Prometheus's
rate()andincrease()handle the reset. - Shutting down is not a failure. A sync, report, or reconcile cut short because the process is stopping is not counted.
- The endpoint holds only these counters: no Go runtime or process metrics and no
target_info. TheOTEL_*environment variables have no effect on it. - Metrics are best-effort. If the port can't be opened, the process logs a warning and keeps running without metrics.
- Configuration.
METRICS_ENABLED=falseturns metrics off and opens no port.METRICS_ADDRmoves the port; if you change it, change the container port namedmetricsin the manifests to match.
Checking an endpoint
Before you configure Prometheus, check that each endpoint answers. From a machine with access to the cluster:
# ConfigHub server (a cub server install uses the confighub namespace)
kubectl -n confighub port-forward deploy/confighub 9090:9090
curl -s localhost:9090/internal/metrics | grep -E '^(requests_total|usage_active_users_total)'
# argobot
kubectl -n argobot port-forward deploy/argobot 9091:9090
curl -s localhost:9091/metrics
# registrybot
kubectl -n registrybot port-forward deploy/registrybot 9093:9090
curl -s localhost:9093/metrics
Run each port-forward in its own terminal, or in the background. Adjust the namespaces if you installed into others.
Collecting with Prometheus
How you tell Prometheus about the targets depends on how your Prometheus finds them. Pick the section that matches your setup.
Prometheus Operator and kube-prometheus-stack
The Prometheus Operator, which kube-prometheus-stack installs, discovers targets from PodMonitor and ServiceMonitor resources and ignores prometheus.io/* annotations. Create a PodMonitor for each process:
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: confighub
namespace: confighub
labels:
release: kube-prometheus-stack # match your Prometheus's podMonitorSelector
spec:
selector:
matchLabels:
app: confighub
podMetricsEndpoints:
- port: http # the API port only; see above
path: /internal/metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: argobot
namespace: argobot
labels:
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app: argobot
podMetricsEndpoints:
- port: metrics
path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: registrybot
namespace: registrybot
labels:
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app: registrybot
podMetricsEndpoints:
- port: metrics
path: /metrics
The release label is what kube-prometheus-stack's default podMonitorSelector looks for, with the value of your Helm release name. If your Prometheus selects monitors differently, check its selector:
kubectl get prometheus -A -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.podMonitorSelector}{"\n"}{end}'
An empty podMonitorNamespaceSelector limits Prometheus to PodMonitors in its own namespace. If yours is empty, either create the PodMonitors there with a namespaceSelector that names the target namespaces, or widen the selector.
Each target appears under Status > Targets in the Prometheus UI as podMonitor/<namespace>/<name>.
Discovery by annotation
Many Prometheus installations, including the community prometheus Helm chart, scrape any pod that carries the prometheus.io/scrape: "true" annotation, at the port and path named by prometheus.io/port and prometheus.io/path.
- ConfigHub server.
cub server install --enable-metricsadds the annotations to the server's pod, naming the API port and/internal/metrics. A re-run keeps them;--enable-metrics=falseremoves them. - argobot. argobot's installer package adds
prometheus.io/scrapeandprometheus.io/pathunless itsmetricsinput isfalse. It sets noprometheus.io/port, so Prometheus scrapes the container port namedmetrics. The plain manifests in argobot's repository have no annotations; add them to the pod template. - registrybot. Its manifests have no annotations; add them to the pod template.
To add them yourself:
spec:
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: /metrics # /internal/metrics for the ConfigHub server
To check what a pod already carries:
kubectl -n argobot get pods -l app=argobot -o jsonpath='{.items[0].metadata.annotations}'
If your Prometheus doesn't yet honor the annotations, this is the conventional scrape job that does:
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: app
With no prometheus.io/port annotation, the port rewrite doesn't match and Prometheus scrapes every declared container port of the pod. That is why argobot's installer package declares only one port that serves metrics.
Static targets
For a Prometheus outside the cluster, or a bot run outside Kubernetes, list the targets directly:
scrape_configs:
- job_name: confighub
metrics_path: /internal/metrics
static_configs:
- targets: ["confighub.internal.example.com:9090"]
- job_name: argobot
static_configs:
- targets: ["argobot.internal.example.com:9090"]
- job_name: registrybot
static_configs:
- targets: ["registrybot.internal.example.com:9090"]
Each target must be reachable from Prometheus without passing through an ingress that hides /internal. With several replicas of the ConfigHub server, scrape each pod, not a Service that balances between them; otherwise each scrape reaches a different replica and the counters appear to jump.
Collecting with the OpenTelemetry Collector
To send the metrics to a backend that takes OTLP, or to a vendor the OpenTelemetry Collector exports to, have a Collector scrape the endpoints and export what it scrapes.
The processes only serve their metrics to be scraped. None of them pushes OTLP, and none reads the OTEL_EXPORTER_* variables, so the Collector has to pull. Use the Prometheus receiver, which is in the contrib and k8s distributions of the Collector. It takes Prometheus scrape configuration, so the jobs in the previous section work in it, with one change: the Collector expands $ in its configuration as an environment variable, so a $ in a relabel rule, as in $1:$2, has to be written $$.
Scraping in Kubernetes
This configuration finds the pods by Kubernetes discovery and keeps only the port that serves metrics, so it needs no annotations:
receivers:
prometheus:
config:
scrape_configs:
- job_name: confighub
metrics_path: /internal/metrics
kubernetes_sd_configs:
- role: pod
namespaces:
names: [confighub]
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
action: keep
regex: confighub;http # the API port only
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- job_name: argobot
kubernetes_sd_configs:
- role: pod
namespaces:
names: [argobot]
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
action: keep
regex: argobot;metrics
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- job_name: registrybot
kubernetes_sd_configs:
- role: pod
namespaces:
names: [registrybot]
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
action: keep
regex: registrybot;metrics
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
processors:
batch: {}
exporters:
otlphttp:
endpoint: https://otlp.example.com # your backend's OTLP endpoint
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [otlphttp]
Kubernetes discovery needs the Collector's service account to get, list, and watch pods in the namespaces it scrapes. If argobot runs in several clusters, run a Collector in each, or add a label that names the cluster, for example with a resource processor, so the clusters' series stay apart.
Outside Kubernetes, use static_configs as in Static targets.
What the Collector does to the metrics
- The scrape job becomes the service. The receiver sets the
service.nameresource attribute from the job name andservice.instance.idfrom the target's address. The labels the relabel rules add,namespaceandpodabove, stay on each data point. - Metric names are the scraped Prometheus names, such as
argobot_syncs_total, not the OpenTelemetry instrument names such asargobot.syncs. Depending on the Collector's version and its name normalization settings, the receiver may trim the_totaland unit suffixes. Check how a metric appears in your backend before you write dashboards and alerts against it. - Counters arrive as cumulative sums. A process restart resets them, and the receiver marks the reset, so backends that compute rates handle it. If your backend wants delta temporality, add the
cumulativetodeltaprocessor. - The server's
target_infois merged into the resource attributes and not kept as a metric. argobot and registrybot export notarget_info.
The queries and alerting rules below are PromQL, so they work unchanged only in a backend that stores the Prometheus names and speaks PromQL. In other backends, use them as a description of what to compute.
With the OpenTelemetry Operator
The OpenTelemetry Operator's Target Allocator can read the PodMonitors above and hand their targets to the Collector, so you keep one description of the targets whichever system scrapes them. The Operator's documentation covers enabling the Target Allocator and its prometheusCR settings; its PodMonitor selector has to select the PodMonitors' labels, and it needs the Prometheus Operator's CRDs installed.
Useful queries
The queries below select the server's series with namespace="confighub", which the PodMonitor and the annotation job above add. Adjust it to your namespace, or to whatever label identifies the server in your setup.
API error rate, the share of requests answered with a 5xx status:
sum by (server) (rate(requests_total{namespace="confighub", http_response_status_code=~"5.."}[5m]))
/
sum by (server) (rate(requests_total{namespace="confighub"}[5m]))
Busiest routes:
topk(10, sum by (http_route) (rate(requests_total{namespace="confighub", server="api"}[5m])))
95th percentile request latency, by server:
histogram_quantile(0.95,
sum by (le, server) (rate(http_server_request_duration_seconds_bucket{namespace="confighub"}[5m])))
argobot sync failure ratio, per controller:
sum by (controller) (rate(argobot_syncs_total{result="failure"}[15m]))
/
sum by (controller) (rate(argobot_syncs_total[15m]))
Published Releases argobot couldn't match to anything to sync:
increase(argobot_events_unresolved_total[1h])
registrybot reconcile failures:
sum(increase(registrybot_reconciles_total{result="failure"}[30m]))
Alerting rules
These rules are a starting point. Tune the thresholds and durations to your traffic. As a plain Prometheus rules file:
groups:
- name: confighub
rules:
- alert: ConfigHubMetricsDown
expr: up{namespace=~"confighub|argobot|registrybot"} == 0
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.namespace }}/{{ $labels.pod }} has not been scraped for 5 minutes"
- alert: ConfigHubAPIErrors
expr: |
sum(rate(requests_total{namespace="confighub", server="api", http_response_status_code=~"5.."}[5m]))
/
sum(rate(requests_total{namespace="confighub", server="api"}[5m]))
> 0.05
for: 10m
labels:
severity: warning
annotations:
summary: "More than 5% of ConfigHub API requests are failing"
- alert: ArgobotSyncFailing
expr: sum by (namespace, pod, controller) (increase(argobot_syncs_total{result="failure"}[15m])) > 0
for: 15m
labels:
severity: warning
annotations:
summary: "argobot cannot sync {{ $labels.controller }}; new Releases wait for the controller's next poll"
- alert: ArgobotStatusReportsFailing
expr: sum by (namespace, pod, controller) (increase(argobot_status_reports_total{result="failure"}[15m])) > 0
for: 30m
labels:
severity: info
annotations:
summary: "argobot cannot report live status to ConfigHub"
- alert: RegistrybotReconcileFailing
expr: sum by (namespace, pod) (increase(registrybot_reconciles_total{result="failure"}[30m])) > 0
for: 30m
labels:
severity: warning
annotations:
summary: "registrybot cannot update fact Units; check its /status endpoint"
With the Prometheus Operator, put the same groups under the spec of a PrometheusRule, labeled to match your Prometheus's ruleSelector (for kube-prometheus-stack, release: <release name>):
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: confighub
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
groups:
- name: confighub
rules:
# the rules above
Further reading
- Self-hosting ConfigHub Enterprise
- Integrating with GitOps operators, where argobot fits
- Updating container images automatically, where registrybot fits
- argobot's README and registrybot's README