Resource Profile
Apiserver does not process high-volume telemetry directly. It handles UI and API requests, reads and writes metadata in PostgreSQL, queries telemetry from ClickHouse, and starts longer-running work in Temporal. Because most durable and expensive work happens in PostgreSQL, ClickHouse, and Temporal, Apiserver should usually stay low CPU and memory. Short request spikes are expected, but sustained high CPU, memory, or throttling is not the normal operating mode.Autoscaling
Leave Apiserver autoscaling enabled unless Metoro support recommends otherwise. The default chart values run multiple replicas and let Kubernetes adjust replica count as UI and API traffic changes. Manual changes to Apiserver CPU, memory, or replica bounds should be rare. Make them only when there is sustained Apiserver-specific pressure, such as persistent CPU throttling, OOM restarts, or request latency that is isolated to Apiserver itself.Restarts And Availability
Apiserver does not own durable local state. Pods can be killed, restarted, or recreated without losing application data. Kubernetes readiness, rolling updates, and multiple replicas keep the UI and API path available while individual pods are replaced. If all Apiserver pods are unavailable, users and exporters cannot reach the control plane, but data is not lost from Apiserver local storage because there is no durable local storage.Behavior During A ClickHouse Outage
Apiserver starts and stays Ready even when ClickHouse is unreachable. Authentication, settings, and other PostgreSQL-backed endpoints keep working, while telemetry queries return a clear “clickhouse is unavailable” error until connectivity is restored. The ClickHouse bootstrap (custom functions, cache tables, workload definitions) retries in the background and completes automatically once ClickHouse comes back, with no Apiserver restart needed. Because pods no longer crashloop when ClickHouse is down or misconfigured (including bad ClickHouse credentials), monitor ClickHouse availability directly rather than relying on Apiserver pod health:/api/v1/healthreportsdependencies.clickhousewithstatus(availableorunavailable), whether the bootstrap has completed, and the last error.- The
metoro.apiserver.clickhouse_readygauge is 1 when ClickHouse is reachable and 0 while Apiserver is serving degraded. Alert when it stays 0 for more than a few minutes.
Telemetry Errors Right After A Fresh Install
12.x installations serve every telemetry query from Metoro’s new-generation ClickHouse tables, which the Ingester creates on its first boot. Table creation runs through a serial DDL queue and can take a few minutes on a fresh cluster, so the Apiserver usually comes up before the tables exist. During that window, telemetry pages (logs, traces, metrics, Kubernetes views, profiles) return errors while the rest of the UI works. This window heals itself. The Apiserver always routes telemetry queries at the new-generation tables, so queries fail transiently while the tables are being created, then start succeeding as soon as the Ingester’s DDL completes, with no restart needed.Webhook Secret Store
MetoroWebhook resources can reference Kubernetes Secrets for header and body values (headersFrom / bodyFrom, see Webhooks). The exporter on each monitored cluster reads only the referenced Secret keys and uploads them to the Apiserver, which stores them encrypted in PostgreSQL and resolves them in memory at delivery time.
The store is off by default: uploads are refused and any webhook that carries Secret references fails closed (it is never sent without its auth header). To turn it on, create a Kubernetes Secret holding a base64-encoded 32-byte key in the hub namespace (the namespace the metoro-onprem release runs in, metoro-hub in the installation guide; substitute yours if it differs) and point the chart at it:
metoro-exporter-onprem values:
get on the specific Secrets that Metoro resources reference (it never lists or watches Secrets), and by default it is granted that get in its own release namespace only. Anyone who can create a Metoro resource with a secretKeyRef in an in-scope namespace can have the exporter read the named Secret there, so keep the namespace list as narrow as possible.
Before Tuning Apiserver
If users report slow UI or API behavior, check dependencies before changing Apiserver resources:- ClickHouse query latency, CPU pressure, memory pressure, and replica health.
- PostgreSQL metadata latency, connection pressure, and primary/standby health.
- Temporal workflow health for background features.
- Ingress, load balancer, DNS, and TLS behavior on the user-facing path.
- Kubernetes node capacity, scheduling events, and pod restarts.
