Skip to main content
Apiserver serves the Metoro UI, public API, and exporter control-plane API. It is a stateless service layer and normally should not need manual resource tuning.

Resource Profile

Apiserver does not process high-volume telemetry directly. It handles UI and API requests, reads and writes metadata in PostgreSQL, queries telemetry from ClickHouse, and starts longer-running work in Temporal. Because most durable and expensive work happens in PostgreSQL, ClickHouse, and Temporal, Apiserver should usually stay low CPU and memory. Short request spikes are expected, but sustained high CPU, memory, or throttling is not the normal operating mode.

Autoscaling

Leave Apiserver autoscaling enabled unless Metoro support recommends otherwise. The default chart values run multiple replicas and let Kubernetes adjust replica count as UI and API traffic changes. Manual changes to Apiserver CPU, memory, or replica bounds should be rare. Make them only when there is sustained Apiserver-specific pressure, such as persistent CPU throttling, OOM restarts, or request latency that is isolated to Apiserver itself.

Restarts And Availability

Apiserver does not own durable local state. Pods can be killed, restarted, or recreated without losing application data. Kubernetes readiness, rolling updates, and multiple replicas keep the UI and API path available while individual pods are replaced. If all Apiserver pods are unavailable, users and exporters cannot reach the control plane, but data is not lost from Apiserver local storage because there is no durable local storage.

Behavior During A ClickHouse Outage

Apiserver starts and stays Ready even when ClickHouse is unreachable. Authentication, settings, and other PostgreSQL-backed endpoints keep working, while telemetry queries return a clear “clickhouse is unavailable” error until connectivity is restored. The ClickHouse bootstrap (custom functions, cache tables, workload definitions) retries in the background and completes automatically once ClickHouse comes back, with no Apiserver restart needed. Because pods no longer crashloop when ClickHouse is down or misconfigured (including bad ClickHouse credentials), monitor ClickHouse availability directly rather than relying on Apiserver pod health:
  • /api/v1/health reports dependencies.clickhouse with status (available or unavailable), whether the bootstrap has completed, and the last error.
  • The metoro.apiserver.clickhouse_ready gauge is 1 when ClickHouse is reachable and 0 while Apiserver is serving degraded. Alert when it stays 0 for more than a few minutes.

Telemetry Errors Right After A Fresh Install

12.x installations serve every telemetry query from Metoro’s new-generation ClickHouse tables, which the Ingester creates on its first boot. Table creation runs through a serial DDL queue and can take a few minutes on a fresh cluster, so the Apiserver usually comes up before the tables exist. During that window, telemetry pages (logs, traces, metrics, Kubernetes views, profiles) return errors while the rest of the UI works. This window heals itself. The Apiserver always routes telemetry queries at the new-generation tables, so queries fail transiently while the tables are being created, then start succeeding as soon as the Ingester’s DDL completes, with no restart needed.

Webhook Secret Store

MetoroWebhook resources can reference Kubernetes Secrets for header and body values (headersFrom / bodyFrom, see Webhooks). The exporter on each monitored cluster reads only the referenced Secret keys and uploads them to the Apiserver, which stores them encrypted in PostgreSQL and resolves them in memory at delivery time. The store is off by default: uploads are refused and any webhook that carries Secret references fails closed (it is never sent without its auth header). To turn it on, create a Kubernetes Secret holding a base64-encoded 32-byte key in the hub namespace (the namespace the metoro-onprem release runs in, metoro-hub in the installation guide; substitute yours if it differs) and point the chart at it:
Back the key up together with your PostgreSQL backups and never rotate it by overwriting the Secret. The chart deliberately never generates this key: if it changes, every stored value becomes permanently undecryptable and affected webhooks fail until their Secrets are re-mirrored.
On each monitored cluster, the exporter also needs permission to read the referenced Secrets. Enable it in the metoro-exporter-onprem values:
The exporter only ever issues get on the specific Secrets that Metoro resources reference (it never lists or watches Secrets), and by default it is granted that get in its own release namespace only. Anyone who can create a Metoro resource with a secretKeyRef in an in-scope namespace can have the exporter read the named Secret there, so keep the namespace list as narrow as possible.

Before Tuning Apiserver

If users report slow UI or API behavior, check dependencies before changing Apiserver resources:
  • ClickHouse query latency, CPU pressure, memory pressure, and replica health.
  • PostgreSQL metadata latency, connection pressure, and primary/standby health.
  • Temporal workflow health for background features.
  • Ingress, load balancer, DNS, and TLS behavior on the user-facing path.
  • Kubernetes node capacity, scheduling events, and pod restarts.
Tune Apiserver only after the signal clearly points to Apiserver rather than one of its dependencies.