Upgrade 10.0.1 to 10.0.2 - Metoro Documentation

Metoro 10.0.2 is a patch release in the 10.x on-premises line. It keeps the 10.x Helm values API and works around a ClickHouse operator bootstrap edge case by creating Metoro tables with stable identifiers. This prevents replicas that are reconciled at different times from creating different replication metadata for the same logical tables. Use this page for existing metoro-onprem 10.0.1 hubs. If you are installing a new hub, install 10.0.2 directly.

These commands assume the Helm release is named metoro and runs in the metoro-hub namespace. Adjust the release name and namespace if your installation uses different values.

Choose The Upgrade Path

Current deployment	Recommended path
External ClickHouse	Run the normal Helm upgrade. Do not delete ClickHouse storage from the hub cluster.
Bundled ClickHouse with healthy replicas	Run the normal Helm upgrade. Existing table metadata is left in place.
Bundled ClickHouse from a 10.0.1 install where Metoro Support has identified the ClickHouse operator bootstrap edge case	Reset bundled ClickHouse storage, then install 10.0.2.
New install or disposable test install	Install 10.0.2 directly.

The 10.0.2 workaround applies when tables are created. A normal upgrade does not rewrite table metadata that already exists. If Metoro Support has identified an early bundled ClickHouse replica metadata mismatch caused during operator bootstrap, the supported remediation is to reset bundled ClickHouse storage and recreate it on 10.0.2.

Normal Helm Upgrade

Use this path unless Metoro Support has told you to reset bundled ClickHouse state. Start from the values file used for the existing hub release. If you do not have it locally, export the user-supplied Helm values:

helm -n metoro-hub get values metoro > metoro-hub-values.yaml

Upgrade the hub chart to 10.0.2:

helm upgrade --install metoro oci://quay.io/metoro/charts/metoro-onprem \
  --namespace metoro-hub \
  --version 10.0.2 \
  --values metoro-hub-values.yaml

Check the rollout:

helm -n metoro-hub status metoro
kubectl -n metoro-hub rollout status deployment/apiserver
kubectl -n metoro-hub rollout status deployment/ingester
kubectl -n metoro-hub get pods -l app=metoro-clickhouse
kubectl -n metoro-hub get pods -l app=metoro-clickhouse-keeper

Expected outcome:

Apiserver and Ingester roll to the 10.0.2 images.
Bundled ClickHouse and Keeper remain ready, or briefly reconcile one pod at a time.
Existing telemetry remains in ClickHouse.
New tables created after the upgrade use stable table identifiers and avoid the operator bootstrap edge case.

Reset Bundled ClickHouse

Use this path only for bundled ClickHouse deployments where Metoro Support has confirmed that the existing 10.0.1 ClickHouse replica metadata should be discarded.

This reset deletes telemetry stored in bundled ClickHouse. It does not delete PostgreSQL application metadata unless you also delete PostgreSQL resources. Do not use this path for external ClickHouse, or for any deployment where existing telemetry must be preserved.

Before starting, confirm that you have:

The current metoro-hub-values.yaml.
Access to the same image pull, auth, ingress, object storage, and database Secrets used by the current release.
A maintenance window for ClickHouse-backed queries and ingestion. Ingester may reject or drop telemetry while ClickHouse is being recreated.

1. Upgrade Hub Services To 10.0.2

Upgrade the hub to 10.0.2 before deleting ClickHouse. This ensures the ingester has the 10.0.2 schema bootstrap behavior before any fresh bundled ClickHouse tables can be created.

helm upgrade --install metoro oci://quay.io/metoro/charts/metoro-onprem \
  --namespace metoro-hub \
  --version 10.0.2 \
  --values metoro-hub-values.yaml

Wait for Apiserver and Ingester to finish rolling out:

kubectl -n metoro-hub rollout status deployment/apiserver
kubectl -n metoro-hub rollout status deployment/ingester

Do not continue until the rollouts have completed. A 10.0.1 ingester must not be able to initialize tables in the recreated ClickHouse cluster.

2. Delete Bundled ClickHouse Resources

This will cause the apiserver and ingester to fail while clickhouse is being removed and reinstalled. Additionally, telemetry data will be deleted.

kubectl -n metoro-hub delete ClickHouseInstallation metoro --wait=true
kubectl -n metoro-hub delete ClickHouseKeeperInstallation clickhouse-metoro --wait=true

List the ClickHouse and Keeper PVCs and their bound PVs:

kubectl -n metoro-hub get pvc \
  | grep -E '(chi-metoro|chk-clickhouse-metoro)'

Delete the ClickHouse and Keeper PVCs shown by the previous command: Do not delete PVCs whose names belong to PostgreSQL, Temporal, or other hub services.

kubectl -n metoro-hub delete pvc PVC_1_CHANGE_ME PVC_2_CHANGE_ME PVC_..._CHANGE_ME

If the bound PVs still exist after deleting the PVCs, delete those ClickHouse and Keeper PVs before continuing:

kubectl -n metoro-hub get pv \
  | grep -E '(chi-metoro|chk-clickhouse-metoro)'

kubectl delete pv PV_1_CHANGE_ME PV_2_CHANGE_ME PV_..._CHANGE_ME

If the StorageClass uses Retain, deleting the Kubernetes PV object may still leave the backing disk or volume in the storage provider. Those retained ClickHouse and Keeper volumes must not be rebound into the recreated 10.0.2 ClickHouse installation. Delete or quarantine the backing volumes according to your cluster storage process.

3. Recreate Bundled ClickHouse On 10.0.2

Run the same 10.0.2 Helm upgrade again. The second invocation recreates the ClickHouse and Keeper resources that were deleted in the previous step, while the already-upgraded 10.0.2 Ingester remains in place.

helm upgrade --install metoro oci://quay.io/metoro/charts/metoro-onprem \
  --namespace metoro-hub \
  --version 10.0.2 \
  --values metoro-hub-values.yaml

4. Verify The Reset

Wait for the hub services, ClickHouse, and Keeper to become ready:

kubectl -n metoro-hub rollout status deployment/apiserver
kubectl -n metoro-hub rollout status deployment/ingester
kubectl -n metoro-hub get pods -l app=metoro-clickhouse
kubectl -n metoro-hub get pods -l app=metoro-clickhouse-keeper

Before proceeding to the next step, ensure that all chi replicas are up and running.

5. Restart Hub And Exporter Workloads

Restart Apiserver and Ingester so they reconnect to the recreated bundled ClickHouse cluster:

kubectl -n metoro-hub rollout restart deployment/apiserver deployment/ingester
kubectl -n metoro-hub rollout status deployment/apiserver
kubectl -n metoro-hub rollout status deployment/ingester

If hub self-monitoring is enabled, restart the hub self-monitoring exporter:

kubectl -n metoro-hub rollout restart deployment/metoro-exporter
kubectl -n metoro-hub rollout status deployment/metoro-exporter

Verify ClickHouse Tables

For bundled ClickHouse deployments, run this read-only check after the upgrade path completes. It execs into one ClickHouse pod and prints each Metoro replicated table, the table UUIDs seen, the number of replicas seen, and the number of Keeper paths seen.

CH_POD="$(kubectl -n metoro-hub get pod -l app=metoro-clickhouse -o jsonpath='{.items[0].metadata.name}')"

kubectl -n metoro-hub exec "$CH_POD" -- clickhouse-client --database default --query "
SELECT
    table,
    arrayStringConcat(groupUniqArray(toString(uuid)), ',') AS uuid,
    count() AS replicas_seen,
    uniqExact(zookeeper_path) AS keeper_paths
FROM clusterAllReplicas('metoro', system, replicas)
WHERE database = currentDatabase()
  AND startsWith(table, 'metoro_')
GROUP BY table
ORDER BY table
FORMAT PrettyCompact
"

Example output:

   ┌─table────────────────────────────────┬─uuid─────────────────────────────────┬─replicas_seen─┬─keeper_paths─┐
│ metoro_k8s_events                    │ 0d425ad5-0de8-4c4f-8783-97735a6f9067 │             3 │            1 │
│ metoro_k8s_resource_liveness_v2      │ f85884bf-dd30-442b-83a6-59ac5f02359d │             3 │            1 │
│ metoro_k8s_resources_v2              │ c7220182-de0c-45f4-9611-0000694e8069 │             3 │            1 │
│ metoro_otel_logs_v2                  │ 6d4dd701-5b16-4762-adb3-5bf48ea7915f │             3 │            1 │
│ metoro_otel_metrics_v3               │ 50c07bb8-acfd-4416-927b-5cf691dce536 │             3 │            1 │
│ metoro_otel_profiles                 │ ca76451d-8a25-42d1-82c3-803e495b472d │             3 │            1 │
│ metoro_otel_traces_v2                │ 951d431a-65df-420a-b2fa-ac027a3d68cf │             3 │            1 │
│ metoro_otel_traces_v2_time_optimized │ b2f0974d-7f91-4671-9ea4-2c0f68f9e6fd │             3 │            1 │
│ metoro_usage_v2                      │ 32036875-4c45-4d10-82f2-9948fc6f60ce │             3 │            1 │
   └──────────────────────────────────────┴──────────────────────────────────────┴───────────────┴──────────────┘

For the default bundled ClickHouse deployment, each table should show one uuid, replicas_seen as 3, and keeper_paths as 1. If clickhouse.bundled.replicaCount is not 3, replicas_seen should match the configured replica count.

​Choose The Upgrade Path

​Normal Helm Upgrade

​Reset Bundled ClickHouse

​1. Upgrade Hub Services To 10.0.2

​2. Delete Bundled ClickHouse Resources

​3. Recreate Bundled ClickHouse On 10.0.2

​4. Verify The Reset

​5. Restart Hub And Exporter Workloads

​Verify ClickHouse Tables

Choose The Upgrade Path

Normal Helm Upgrade

Reset Bundled ClickHouse

1. Upgrade Hub Services To 10.0.2

2. Delete Bundled ClickHouse Resources

3. Recreate Bundled ClickHouse On 10.0.2

4. Verify The Reset

5. Restart Hub And Exporter Workloads

Verify ClickHouse Tables