> ## Documentation Index
> Fetch the complete documentation index at: https://metoro.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Apiserver Operations

> Operate the Metoro Hub Apiserver in on-premises deployments

Apiserver serves the Metoro UI, public API, and exporter control-plane API. It is a stateless service layer and normally should not need manual resource tuning.

## Resource Profile

Apiserver does not process high-volume telemetry directly. It handles UI and API requests, reads and writes metadata in PostgreSQL, queries telemetry from ClickHouse, and starts longer-running work in Temporal.

Because most durable and expensive work happens in PostgreSQL, ClickHouse, and Temporal, Apiserver should usually stay low CPU and memory. Short request spikes are expected, but sustained high CPU, memory, or throttling is not the normal operating mode.

## Autoscaling

Leave Apiserver autoscaling enabled unless Metoro support recommends otherwise. The default chart values run multiple replicas and let Kubernetes adjust replica count as UI and API traffic changes.

Manual changes to Apiserver CPU, memory, or replica bounds should be rare. Make them only when there is sustained Apiserver-specific pressure, such as persistent CPU throttling, OOM restarts, or request latency that is isolated to Apiserver itself.

## Restarts And Availability

Apiserver does not own durable local state. Pods can be killed, restarted, or recreated without losing application data.

Kubernetes readiness, rolling updates, and multiple replicas keep the UI and API path available while individual pods are replaced. If all Apiserver pods are unavailable, users and exporters cannot reach the control plane, but data is not lost from Apiserver local storage because there is no durable local storage.

## Behavior During A ClickHouse Outage

Apiserver starts and stays Ready even when ClickHouse is unreachable. Authentication, settings, and other PostgreSQL-backed endpoints keep working, while telemetry queries return a clear "clickhouse is unavailable" error until connectivity is restored. The ClickHouse bootstrap (custom functions, cache tables, workload definitions) retries in the background and completes automatically once ClickHouse comes back, with no Apiserver restart needed.

Because pods no longer crashloop when ClickHouse is down or misconfigured (including bad ClickHouse credentials), monitor ClickHouse availability directly rather than relying on Apiserver pod health:

* `/api/v1/health` reports `dependencies.clickhouse` with `status` (`available` or `unavailable`), whether the bootstrap has completed, and the last error.
* The `metoro.apiserver.clickhouse_ready` gauge is 1 when ClickHouse is reachable and 0 while Apiserver is serving degraded. Alert when it stays 0 for more than a few minutes.

## Telemetry Errors Right After A Fresh Install

11.x installations serve every telemetry query from Metoro's new-generation ClickHouse tables, which the Ingester creates on its first boot. Table creation runs through a serial DDL queue and can take a few minutes on a fresh cluster, so the Apiserver usually comes up before the tables exist. During that window, telemetry pages (logs, traces, metrics, Kubernetes views, profiles) return errors while the rest of the UI works.

This window heals itself. The chart marks all telemetry signals as new-generation-only (`METORO_TABLES_V3_READS_AUTHORITATIVE`), so the Apiserver keeps routing queries at the new tables while they are still being created: queries fail transiently, then start succeeding as soon as the Ingester's DDL completes, with no restart needed. The waiting state is visible in the Apiserver logs as `... tables are confirmed missing, but reads are authoritative for the family ... KEEPING routing`; on a healthy boot the Apiserver instead logs `Tables v3 read routing enabled for family` for all six telemetry families.

## Before Tuning Apiserver

If users report slow UI or API behavior, check dependencies before changing Apiserver resources:

* ClickHouse query latency, CPU pressure, memory pressure, and replica health.
* PostgreSQL metadata latency, connection pressure, and primary/standby health.
* Temporal workflow health for background features.
* Ingress, load balancer, DNS, and TLS behavior on the user-facing path.
* Kubernetes node capacity, scheduling events, and pod restarts.

Tune Apiserver only after the signal clearly points to Apiserver rather than one of its dependencies.