> ## Documentation Index
> Fetch the complete documentation index at: https://metoro.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade 10.0.1 to 10.0.2

> Upgrade Metoro on-premises 10.0.1 hubs to 10.0.2 and reset early bundled ClickHouse installs when required

Metoro 10.0.2 is a patch release in the 10.x on-premises line. It keeps the 10.x Helm values API and works around a ClickHouse operator bootstrap edge case by creating Metoro tables with stable identifiers. This prevents replicas that are reconciled at different times from creating different replication metadata for the same logical tables.

Use this page for existing `metoro-onprem` 10.0.1 hubs. If you are installing a new hub, install `10.0.2` directly.

<Info>
  These commands assume the Helm release is named `metoro` and runs in the `metoro-hub` namespace. Adjust the release name and namespace if your installation uses different values.
</Info>

## Choose The Upgrade Path

| Current deployment                                                                                                       | Recommended path                                                                    |
| ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- |
| External ClickHouse                                                                                                      | Run the normal Helm upgrade. Do not delete ClickHouse storage from the hub cluster. |
| Bundled ClickHouse with healthy replicas                                                                                 | Run the normal Helm upgrade. Existing table metadata is left in place.              |
| Bundled ClickHouse from a 10.0.1 install where Metoro Support has identified the ClickHouse operator bootstrap edge case | Reset bundled ClickHouse storage, then install 10.0.2.                              |
| New install or disposable test install                                                                                   | Install 10.0.2 directly.                                                            |

The 10.0.2 workaround applies when tables are created. A normal upgrade does not rewrite table metadata that already exists. If Metoro Support has identified an early bundled ClickHouse replica metadata mismatch caused during operator bootstrap, the supported remediation is to reset bundled ClickHouse storage and recreate it on 10.0.2.

## Normal Helm Upgrade

Use this path unless Metoro Support has told you to reset bundled ClickHouse state.

Start from the values file used for the existing hub release. If you do not have it locally, export the user-supplied Helm values:

```bash theme={null}
helm -n metoro-hub get values metoro > metoro-hub-values.yaml
```

Upgrade the hub chart to `10.0.2`:

```bash theme={null}
helm upgrade --install metoro oci://quay.io/metoro/charts/metoro-onprem \
  --namespace metoro-hub \
  --version 10.0.2 \
  --values metoro-hub-values.yaml
```

Check the rollout:

```bash theme={null}
helm -n metoro-hub status metoro
kubectl -n metoro-hub rollout status deployment/apiserver
kubectl -n metoro-hub rollout status deployment/ingester
kubectl -n metoro-hub get pods -l app=metoro-clickhouse
kubectl -n metoro-hub get pods -l app=metoro-clickhouse-keeper
```

Expected outcome:

* Apiserver and Ingester roll to the 10.0.2 images.
* Bundled ClickHouse and Keeper remain ready, or briefly reconcile one pod at a time.
* Existing telemetry remains in ClickHouse.
* New tables created after the upgrade use stable table identifiers and avoid the operator bootstrap edge case.

## Reset Bundled ClickHouse

Use this path only for bundled ClickHouse deployments where Metoro Support has confirmed that the existing 10.0.1 ClickHouse replica metadata should be discarded.

<Warning>
  This reset deletes telemetry stored in bundled ClickHouse. It does not delete PostgreSQL application metadata unless you also delete PostgreSQL resources. Do not use this path for external ClickHouse, or for any deployment where existing telemetry must be preserved.
</Warning>

Before starting, confirm that you have:

* The current `metoro-hub-values.yaml`.
* Access to the same image pull, auth, ingress, object storage, and database Secrets used by the current release.
* A maintenance window for ClickHouse-backed queries and ingestion. Ingester may reject or drop telemetry while ClickHouse is being recreated.

### 1. Upgrade Hub Services To 10.0.2

Upgrade the hub to `10.0.2` before deleting ClickHouse. This ensures the ingester has the 10.0.2 schema bootstrap behavior before any fresh bundled ClickHouse tables can be created.

```bash theme={null}
helm upgrade --install metoro oci://quay.io/metoro/charts/metoro-onprem \
  --namespace metoro-hub \
  --version 10.0.2 \
  --values metoro-hub-values.yaml
```

Wait for Apiserver and Ingester to finish rolling out:

```bash theme={null}
kubectl -n metoro-hub rollout status deployment/apiserver
kubectl -n metoro-hub rollout status deployment/ingester
```

Do not continue until the rollouts have completed. A 10.0.1 ingester must not be able to initialize tables in the recreated ClickHouse cluster.

### 2. Delete Bundled ClickHouse Resources

<Warning>
  This will cause the apiserver and ingester to fail while clickhouse is being removed and reinstalled. Additionally, telemetry data will be deleted.
</Warning>

```bash theme={null}
kubectl -n metoro-hub delete ClickHouseInstallation metoro --wait=true
kubectl -n metoro-hub delete ClickHouseKeeperInstallation clickhouse-metoro --wait=true
```

List the ClickHouse and Keeper PVCs and their bound PVs:

```bash theme={null}
kubectl -n metoro-hub get pvc \
  | grep -E '(chi-metoro|chk-clickhouse-metoro)'
```

Delete the ClickHouse and Keeper PVCs shown by the previous command:
Do not delete PVCs whose names belong to PostgreSQL, Temporal, or other hub services.

```bash theme={null}
kubectl -n metoro-hub delete pvc PVC_1_CHANGE_ME PVC_2_CHANGE_ME PVC_..._CHANGE_ME
```

If the bound PVs still exist after deleting the PVCs, delete those ClickHouse and Keeper PVs before continuing:

```bash theme={null}
kubectl -n metoro-hub get pv \
  | grep -E '(chi-metoro|chk-clickhouse-metoro)'
```

```bash theme={null}
kubectl delete pv PV_1_CHANGE_ME PV_2_CHANGE_ME PV_..._CHANGE_ME
```

If the StorageClass uses `Retain`, deleting the Kubernetes PV object may still leave the backing disk or volume in the storage provider. Those retained ClickHouse and Keeper volumes must not be rebound into the recreated 10.0.2 ClickHouse installation. Delete or quarantine the backing volumes according to your cluster storage process.

### 3. Recreate Bundled ClickHouse On 10.0.2

Run the same `10.0.2` Helm upgrade again. The second invocation recreates the ClickHouse and Keeper resources that were deleted in the previous step, while the already-upgraded 10.0.2 Ingester remains in place.

```bash theme={null}
helm upgrade --install metoro oci://quay.io/metoro/charts/metoro-onprem \
  --namespace metoro-hub \
  --version 10.0.2 \
  --values metoro-hub-values.yaml
```

### 4. Verify The Reset

Wait for the hub services, ClickHouse, and Keeper to become ready:

```bash theme={null}
kubectl -n metoro-hub rollout status deployment/apiserver
kubectl -n metoro-hub rollout status deployment/ingester
kubectl -n metoro-hub get pods -l app=metoro-clickhouse
kubectl -n metoro-hub get pods -l app=metoro-clickhouse-keeper
```

Before proceeding to the next step, ensure that all chi replicas are up and running.

### 5. Restart Hub And Exporter Workloads

Restart Apiserver and Ingester so they reconnect to the recreated bundled ClickHouse cluster:

```bash theme={null}
kubectl -n metoro-hub rollout restart deployment/apiserver deployment/ingester
kubectl -n metoro-hub rollout status deployment/apiserver
kubectl -n metoro-hub rollout status deployment/ingester
```

If hub self-monitoring is enabled, restart the hub self-monitoring exporter:

```bash theme={null}
kubectl -n metoro-hub rollout restart deployment/metoro-exporter
kubectl -n metoro-hub rollout status deployment/metoro-exporter
```

## Verify ClickHouse Tables

For bundled ClickHouse deployments, run this read-only check after the upgrade path completes. It execs into one ClickHouse pod and prints each Metoro replicated table, the table UUIDs seen, the number of replicas seen, and the number of Keeper paths seen.

```bash theme={null}
CH_POD="$(kubectl -n metoro-hub get pod -l app=metoro-clickhouse -o jsonpath='{.items[0].metadata.name}')"

kubectl -n metoro-hub exec "$CH_POD" -- clickhouse-client --database default --query "
SELECT
    table,
    arrayStringConcat(groupUniqArray(toString(uuid)), ',') AS uuid,
    count() AS replicas_seen,
    uniqExact(zookeeper_path) AS keeper_paths
FROM clusterAllReplicas('metoro', system, replicas)
WHERE database = currentDatabase()
  AND startsWith(table, 'metoro_')
GROUP BY table
ORDER BY table
FORMAT PrettyCompact
"
```

Example output:

```text theme={null}
   ┌─table────────────────────────────────┬─uuid─────────────────────────────────┬─replicas_seen─┬─keeper_paths─┐
1. │ metoro_k8s_events                    │ 0d425ad5-0de8-4c4f-8783-97735a6f9067 │             3 │            1 │
2. │ metoro_k8s_resource_liveness_v2      │ f85884bf-dd30-442b-83a6-59ac5f02359d │             3 │            1 │
3. │ metoro_k8s_resources_v2              │ c7220182-de0c-45f4-9611-0000694e8069 │             3 │            1 │
4. │ metoro_otel_logs_v2                  │ 6d4dd701-5b16-4762-adb3-5bf48ea7915f │             3 │            1 │
5. │ metoro_otel_metrics_v3               │ 50c07bb8-acfd-4416-927b-5cf691dce536 │             3 │            1 │
6. │ metoro_otel_profiles                 │ ca76451d-8a25-42d1-82c3-803e495b472d │             3 │            1 │
7. │ metoro_otel_traces_v2                │ 951d431a-65df-420a-b2fa-ac027a3d68cf │             3 │            1 │
8. │ metoro_otel_traces_v2_time_optimized │ b2f0974d-7f91-4671-9ea4-2c0f68f9e6fd │             3 │            1 │
9. │ metoro_usage_v2                      │ 32036875-4c45-4d10-82f2-9948fc6f60ce │             3 │            1 │
   └──────────────────────────────────────┴──────────────────────────────────────┴───────────────┴──────────────┘
```

For the default bundled ClickHouse deployment, each table should show one `uuid`, `replicas_seen` as `3`, and `keeper_paths` as `1`. If `clickhouse.bundled.replicaCount` is not `3`, `replicas_seen` should match the configured replica count.