Skip to content

Monitoring with Metrics

ConfigHub's processes record their metrics with OpenTelemetry and serve them in the Prometheus text format, through the OpenTelemetry Prometheus exporter. Any system that can scrape a Prometheus endpoint can collect them, including Prometheus itself and the OpenTelemetry Collector's Prometheus receiver. This page lists what each process exposes and how to collect it with Prometheus or the OpenTelemetry Collector.

The exporter turns each OpenTelemetry instrument name into a Prometheus metric name: dots become underscores, counters get a _total suffix, and a unit adds a suffix such as _seconds or _bytes. The instrument argobot.syncs, for example, is exported as argobot_syncs_total, and its attributes become labels. This page uses the exported names, which are the ones you query.

Three processes serve metrics:

Process Where it runs Endpoint Default port
ConfigHub server Your cluster, if you self-host ConfigHub /internal/metrics 9090 (the API port, PORT)
argobot Each cluster it syncs, next to Argo CD, Flux, or Sveltos /metrics 9090 (METRICS_ADDR)
registrybot Wherever you run it /metrics 9090 (METRICS_ADDR)

On the hosted service at hub.confighub.com the server is operated for you and its metrics are not exposed. argobot and registrybot run in your own infrastructure either way, so you can always scrape them.

None of the endpoints require authentication. Keep them inside your network: don't route them through an ingress.

ConfigHub server

The server serves its metrics at /internal/metrics on the API port, 9090 by default. The OCI registry port, 9092 by default, serves the same endpoint, and both return the same metrics: the API's and the registry's requests are recorded together and told apart by the server label. Scrape only the API port; scraping both doubles every series.

The cub server plugin's ingress does not route /internal, so the endpoint is reachable only from inside the cluster. If you deploy from your own manifests, don't route /internal either. It also serves pprof profiles.

HTTP request metrics

Every request to the API server and to the OCI registry is recorded.

Metric Type Labels
requests_total counter server, http_request_method, http_route, http_response_status_code, server_address, url_scheme
http_server_request_duration_seconds histogram server, http_request_method, server_address, url_scheme
http_server_active_requests gauge server, http_request_method, server_address, url_scheme
http_server_request_body_size_bytes histogram same as requests_total
http_server_response_body_size_bytes histogram same as requests_total

The labels are:

  • server: api for the API server, oci for the OCI registry.
  • http_request_method: GET, POST, and so on. Methods outside the standard set are recorded as _OTHER.
  • http_route: the route pattern, such as /api/space/:space_id, not the requested path, so the number of series stays bounded. It is empty when no route matched.
  • http_response_status_code: the status code of the response.
  • server_address: the host the request was sent to, from its Host header. Always empty for the OCI registry, which any client can reach with any Host header.
  • url_scheme: https if the request arrived over TLS or a proxy on a private address said so, otherwise http.

Things to know when you query them:

  • Request duration has no route or status labels. You can break latency down by server and method, but not by route or status code. Durations are measured in whole milliseconds, and the buckets run from 5 ms to 120 s.
  • The size histograms' buckets are 1 KB, 5 KB, 10 KB, 100 KB, 500 KB, 1 MB, 5 MB, 10 MB, 100 MB, and 1 GB.
  • requests_total is a generic name. Other targets may export a metric with the same name, so select the server's series by a label your scrape configuration adds, such as namespace="confighub", or by otel_scope_name="echo", which every HTTP metric of the server carries.

Usage

  • usage_active_users_total (counter): the number of distinct users who have made an authenticated request since the server started. It is kept in memory, so it goes back to zero when the server restarts, and each replica counts on its own.

Runtime and process

The endpoint also carries the Prometheus Go client's standard collectors: Go runtime metrics (go_goroutines, go_memstats_*, go_gc_duration_seconds, and so on) and process metrics (process_cpu_seconds_total, process_resident_memory_bytes, process_open_fds, and so on). OpenTelemetry adds target_info, one series of which carries service_name="confighub".

argobot

argobot serves its metrics at /metrics on METRICS_ADDR, :9090 by default. Every series carries a controller label naming the GitOps controller argobot drives: argocd, flux, or sveltos.

Metric Labels Counts
argobot_syncs_total controller, result Sync requests to the GitOps controller, one per Argo CD Application, Sveltos profile, or Flux OCIRepository a Release event resolved to. result is success or failure.
argobot_events_unresolved_total controller release.published events that resolved to no Application, profile, or OCIRepository, so nothing was synced. A steady rise usually means a sync target is misnamed or its OCI URL names a different Space.
argobot_status_reports_total controller, result Passes of the live status reporter over one watched object. success: the object's Release was found, whether or not anything had to be written. skipped: there was nothing to record on, because the object is gone or names no Space or Release argobot can see. failure: the pass failed and will be retried.

registrybot

registrybot serves its metrics at /metrics on METRICS_ADDR, :9090 by default. This is a separate port from its webhook receiver (:8080), so exposing the webhook does not expose the metrics.

Metric Labels Counts
registrybot_reconciles_total result Reconciles of one repository's fact Unit, from a poll or a webhook. written: the fact Unit got a new revision, including when it was created. unchanged: the facts were already current. failure: the reconcile failed and is left to the next poll. A steady failure rate usually means the GitHub token or the service account's grants are wrong; registrybot's /status endpoint names the repository and the error.
registrybot_discoveries_total result Repositories a signed webhook named that registrybot was not watching. admitted: the discovery policy let the repository in, so it now has a fact Unit. refused: it did not, because discovery is off or the owner or an exclude pattern rules it out.

What argobot and registrybot have in common

  • Every series exists from the start, at zero. A failure series is there before the first failure, so rate() and increase() see the first increment and an alert on it has data to evaluate from the moment the process starts.
  • Counters restart from zero whenever the process does. Prometheus's rate() and increase() handle the reset.
  • Shutting down is not a failure. A sync, report, or reconcile cut short because the process is stopping is not counted.
  • The endpoint holds only these counters: no Go runtime or process metrics and no target_info. The OTEL_* environment variables have no effect on it.
  • Metrics are best-effort. If the port can't be opened, the process logs a warning and keeps running without metrics.
  • Configuration. METRICS_ENABLED=false turns metrics off and opens no port. METRICS_ADDR moves the port; if you change it, change the container port named metrics in the manifests to match.

Checking an endpoint

Before you configure Prometheus, check that each endpoint answers. From a machine with access to the cluster:

# ConfigHub server (a cub server install uses the confighub namespace)
kubectl -n confighub port-forward deploy/confighub 9090:9090
curl -s localhost:9090/internal/metrics | grep -E '^(requests_total|usage_active_users_total)'

# argobot
kubectl -n argobot port-forward deploy/argobot 9091:9090
curl -s localhost:9091/metrics

# registrybot
kubectl -n registrybot port-forward deploy/registrybot 9093:9090
curl -s localhost:9093/metrics

Run each port-forward in its own terminal, or in the background. Adjust the namespaces if you installed into others.

Collecting with Prometheus

How you tell Prometheus about the targets depends on how your Prometheus finds them. Pick the section that matches your setup.

Prometheus Operator and kube-prometheus-stack

The Prometheus Operator, which kube-prometheus-stack installs, discovers targets from PodMonitor and ServiceMonitor resources and ignores prometheus.io/* annotations. Create a PodMonitor for each process:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: confighub
  namespace: confighub
  labels:
    release: kube-prometheus-stack   # match your Prometheus's podMonitorSelector
spec:
  selector:
    matchLabels:
      app: confighub
  podMetricsEndpoints:
    - port: http                # the API port only; see above
      path: /internal/metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: argobot
  namespace: argobot
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: argobot
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: registrybot
  namespace: registrybot
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: registrybot
  podMetricsEndpoints:
    - port: metrics
      path: /metrics

The release label is what kube-prometheus-stack's default podMonitorSelector looks for, with the value of your Helm release name. If your Prometheus selects monitors differently, check its selector:

kubectl get prometheus -A -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.podMonitorSelector}{"\n"}{end}'

An empty podMonitorNamespaceSelector limits Prometheus to PodMonitors in its own namespace. If yours is empty, either create the PodMonitors there with a namespaceSelector that names the target namespaces, or widen the selector.

Each target appears under Status > Targets in the Prometheus UI as podMonitor/<namespace>/<name>.

Discovery by annotation

Many Prometheus installations, including the community prometheus Helm chart, scrape any pod that carries the prometheus.io/scrape: "true" annotation, at the port and path named by prometheus.io/port and prometheus.io/path.

  • ConfigHub server. cub server install --enable-metrics adds the annotations to the server's pod, naming the API port and /internal/metrics. A re-run keeps them; --enable-metrics=false removes them.
  • argobot. argobot's installer package adds prometheus.io/scrape and prometheus.io/path unless its metrics input is false. It sets no prometheus.io/port, so Prometheus scrapes the container port named metrics. The plain manifests in argobot's repository have no annotations; add them to the pod template.
  • registrybot. Its manifests have no annotations; add them to the pod template.

To add them yourself:

spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: /metrics   # /internal/metrics for the ConfigHub server

To check what a pod already carries:

kubectl -n argobot get pods -l app=argobot -o jsonpath='{.items[0].metadata.annotations}'

If your Prometheus doesn't yet honor the annotations, this is the conventional scrape job that does:

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app

With no prometheus.io/port annotation, the port rewrite doesn't match and Prometheus scrapes every declared container port of the pod. That is why argobot's installer package declares only one port that serves metrics.

Static targets

For a Prometheus outside the cluster, or a bot run outside Kubernetes, list the targets directly:

scrape_configs:
  - job_name: confighub
    metrics_path: /internal/metrics
    static_configs:
      - targets: ["confighub.internal.example.com:9090"]
  - job_name: argobot
    static_configs:
      - targets: ["argobot.internal.example.com:9090"]
  - job_name: registrybot
    static_configs:
      - targets: ["registrybot.internal.example.com:9090"]

Each target must be reachable from Prometheus without passing through an ingress that hides /internal. With several replicas of the ConfigHub server, scrape each pod, not a Service that balances between them; otherwise each scrape reaches a different replica and the counters appear to jump.

Collecting with the OpenTelemetry Collector

To send the metrics to a backend that takes OTLP, or to a vendor the OpenTelemetry Collector exports to, have a Collector scrape the endpoints and export what it scrapes.

The processes only serve their metrics to be scraped. None of them pushes OTLP, and none reads the OTEL_EXPORTER_* variables, so the Collector has to pull. Use the Prometheus receiver, which is in the contrib and k8s distributions of the Collector. It takes Prometheus scrape configuration, so the jobs in the previous section work in it, with one change: the Collector expands $ in its configuration as an environment variable, so a $ in a relabel rule, as in $1:$2, has to be written $$.

Scraping in Kubernetes

This configuration finds the pods by Kubernetes discovery and keeps only the port that serves metrics, so it needs no annotations:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: confighub
          metrics_path: /internal/metrics
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: [confighub]
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
              action: keep
              regex: confighub;http      # the API port only
            - source_labels: [__meta_kubernetes_namespace]
              target_label: namespace
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: pod
        - job_name: argobot
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: [argobot]
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
              action: keep
              regex: argobot;metrics
            - source_labels: [__meta_kubernetes_namespace]
              target_label: namespace
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: pod
        - job_name: registrybot
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: [registrybot]
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_app, __meta_kubernetes_pod_container_port_name]
              action: keep
              regex: registrybot;metrics
            - source_labels: [__meta_kubernetes_namespace]
              target_label: namespace
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: pod

processors:
  batch: {}

exporters:
  otlphttp:
    endpoint: https://otlp.example.com   # your backend's OTLP endpoint

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlphttp]

Kubernetes discovery needs the Collector's service account to get, list, and watch pods in the namespaces it scrapes. If argobot runs in several clusters, run a Collector in each, or add a label that names the cluster, for example with a resource processor, so the clusters' series stay apart.

Outside Kubernetes, use static_configs as in Static targets.

What the Collector does to the metrics

  • The scrape job becomes the service. The receiver sets the service.name resource attribute from the job name and service.instance.id from the target's address. The labels the relabel rules add, namespace and pod above, stay on each data point.
  • Metric names are the scraped Prometheus names, such as argobot_syncs_total, not the OpenTelemetry instrument names such as argobot.syncs. Depending on the Collector's version and its name normalization settings, the receiver may trim the _total and unit suffixes. Check how a metric appears in your backend before you write dashboards and alerts against it.
  • Counters arrive as cumulative sums. A process restart resets them, and the receiver marks the reset, so backends that compute rates handle it. If your backend wants delta temporality, add the cumulativetodelta processor.
  • The server's target_info is merged into the resource attributes and not kept as a metric. argobot and registrybot export no target_info.

The queries and alerting rules below are PromQL, so they work unchanged only in a backend that stores the Prometheus names and speaks PromQL. In other backends, use them as a description of what to compute.

With the OpenTelemetry Operator

The OpenTelemetry Operator's Target Allocator can read the PodMonitors above and hand their targets to the Collector, so you keep one description of the targets whichever system scrapes them. The Operator's documentation covers enabling the Target Allocator and its prometheusCR settings; its PodMonitor selector has to select the PodMonitors' labels, and it needs the Prometheus Operator's CRDs installed.

Useful queries

The queries below select the server's series with namespace="confighub", which the PodMonitor and the annotation job above add. Adjust it to your namespace, or to whatever label identifies the server in your setup.

API error rate, the share of requests answered with a 5xx status:

sum by (server) (rate(requests_total{namespace="confighub", http_response_status_code=~"5.."}[5m]))
/
sum by (server) (rate(requests_total{namespace="confighub"}[5m]))

Busiest routes:

topk(10, sum by (http_route) (rate(requests_total{namespace="confighub", server="api"}[5m])))

95th percentile request latency, by server:

histogram_quantile(0.95,
  sum by (le, server) (rate(http_server_request_duration_seconds_bucket{namespace="confighub"}[5m])))

argobot sync failure ratio, per controller:

sum by (controller) (rate(argobot_syncs_total{result="failure"}[15m]))
/
sum by (controller) (rate(argobot_syncs_total[15m]))

Published Releases argobot couldn't match to anything to sync:

increase(argobot_events_unresolved_total[1h])

registrybot reconcile failures:

sum(increase(registrybot_reconciles_total{result="failure"}[30m]))

Alerting rules

These rules are a starting point. Tune the thresholds and durations to your traffic. As a plain Prometheus rules file:

groups:
  - name: confighub
    rules:
      - alert: ConfigHubMetricsDown
        expr: up{namespace=~"confighub|argobot|registrybot"} == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} has not been scraped for 5 minutes"

      - alert: ConfigHubAPIErrors
        expr: |
          sum(rate(requests_total{namespace="confighub", server="api", http_response_status_code=~"5.."}[5m]))
          /
          sum(rate(requests_total{namespace="confighub", server="api"}[5m]))
          > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "More than 5% of ConfigHub API requests are failing"

      - alert: ArgobotSyncFailing
        expr: sum by (namespace, pod, controller) (increase(argobot_syncs_total{result="failure"}[15m])) > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "argobot cannot sync {{ $labels.controller }}; new Releases wait for the controller's next poll"

      - alert: ArgobotStatusReportsFailing
        expr: sum by (namespace, pod, controller) (increase(argobot_status_reports_total{result="failure"}[15m])) > 0
        for: 30m
        labels:
          severity: info
        annotations:
          summary: "argobot cannot report live status to ConfigHub"

      - alert: RegistrybotReconcileFailing
        expr: sum by (namespace, pod) (increase(registrybot_reconciles_total{result="failure"}[30m])) > 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "registrybot cannot update fact Units; check its /status endpoint"

With the Prometheus Operator, put the same groups under the spec of a PrometheusRule, labeled to match your Prometheus's ruleSelector (for kube-prometheus-stack, release: <release name>):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: confighub
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: confighub
      rules:
        # the rules above

Further reading