> ## Documentation Index
> Fetch the complete documentation index at: https://metoro.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Object Storage Cold Tier

> Configure S3-compatible object storage as the ClickHouse cold tier for Metoro on-premises deployments

The bundled ClickHouse can tier telemetry between a fast hot PVC and S3-compatible object storage. Recent data stays on the hot PVC where inserts and merges are fast, and older data moves to object storage, which is cheaper and scales without PVC resizes. This is the recommended production storage model; see [ClickHouse Sizing](/docs/on-premises/11.x/operations/clickhouse/sizing) for how to size each tier.

This page applies to the bundled ClickHouse only. For externally managed ClickHouse, storage tiering is owned by the team operating that service.

## How It Works

When `clickhouse.bundled.objectStorage` is enabled, the chart configures three ClickHouse disks and a `hot_cold` storage policy on every replica:

| Disk          | Backing                                                    | Role                             |
| ------------- | ---------------------------------------------------------- | -------------------------------- |
| `hot_pvc`     | The replica's PVC                                          | Recent data, inserts, and merges |
| `cold_object` | S3-compatible object storage                               | Long-term telemetry              |
| `cold_cache`  | Local disk cache on the PVC (`cacheMaxSize`, default 10Gi) | Caches object-backed reads       |

The Ingester creates every telemetry table with `storage_policy = 'hot_cold'` and a TTL that moves data parts to the cold volume once they are older than `hotStorageDays` (default 1 day). Reads are transparent: queries over older ranges are served through the cache, falling back to object storage on cache misses. Retention deletes continue to work unchanged on both tiers.

<Warning>
  Enable object storage **at install time**. The storage policy and the hot-to-cold TTL are baked into each table's schema when the Ingester first creates it. Enabling object storage on an existing installation configures the disks but leaves already-created tables on their original storage policy, so no data moves to the object store. Migrating an existing installation requires altering each table; contact Metoro support if you need to do this.
</Warning>

## Provision the Bucket and Credentials

Provision an S3-compatible bucket (or a prefix within one) for ClickHouse before installing. The endpoint must be reachable from the hub cluster, and the credentials should be scoped to that bucket or prefix. If the endpoint uses a private CA, include it in the chart's `trustedCAs` material.

Create the credentials Secret in the hub namespace:

```bash theme={null}
kubectl -n metoro-hub create secret generic metoro-clickhouse-object-storage \
  --from-literal=access_key_id="CHANGE_ME_ACCESS_KEY_ID" \
  --from-literal=secret_access_key="CHANGE_ME_SECRET_ACCESS_KEY"
```

## Configure Helm Values

```yaml theme={null}
clickhouse:
  bundled:
    objectStorage:
      enabled: true
      provider: s3
      existingSecret:
        name: metoro-clickhouse-object-storage
      s3:
        # Full URL including the bucket and an optional key prefix, with a
        # trailing slash. All replicas can share one prefix.
        endpoint: https://s3.eu-west-2.amazonaws.com/metoro-clickhouse-cold/clickhouse/
        region: eu-west-2
      # Days of data kept on the hot PVC before parts move to object storage.
      hotStorageDays: 1
      # Per-replica local cache in front of the object disk.
      cacheMaxSize: 10Gi
```

The `endpoint` is passed to ClickHouse's S3 disk as-is, so it must contain the bucket name and any prefix. Path-style URLs (`https://host:port/bucket/prefix/`) work for MinIO and most self-hosted S3 implementations; virtual-hosted style (`https://bucket.s3.region.amazonaws.com/prefix/`) works for AWS. Leave `region` empty for implementations that do not use it.

Azure Blob Storage is also supported with `provider: azureBlob`; it reads `account_name` and `account_key` from the same Secret and takes `azureBlob.storageAccountUrl` and `azureBlob.containerName` instead of the `s3` block.

On startup, each ClickHouse replica verifies it can write to the object store and fails loudly if the endpoint or credentials are wrong, so misconfiguration surfaces immediately rather than at the first TTL move.

## Example: In-Cluster MinIO

For evaluation clusters, or air-gapped environments without an existing object store, an in-cluster [MinIO](https://min.io/) provides an S3-compatible endpoint that exercises the exact same `provider: s3` path. The single-replica setup below is suitable for testing; for production MinIO, follow MinIO's own distributed-deployment guidance.

Create the namespace and root credentials, and give ClickHouse the same credentials:

```bash theme={null}
kubectl create namespace minio

MINIO_USER="metoro-clickhouse"
MINIO_PASS="$(openssl rand -hex 24)"

kubectl -n minio create secret generic minio-root \
  --from-literal=MINIO_ROOT_USER="${MINIO_USER}" \
  --from-literal=MINIO_ROOT_PASSWORD="${MINIO_PASS}"

kubectl -n metoro-hub create secret generic metoro-clickhouse-object-storage \
  --from-literal=access_key_id="${MINIO_USER}" \
  --from-literal=secret_access_key="${MINIO_PASS}"
```

Deploy MinIO with a persistent volume, a Service, and a one-shot Job that creates the bucket:

```yaml theme={null}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-data
  namespace: minio
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 128Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: minio
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: minio
  template:
    metadata:
      labels:
        app.kubernetes.io/name: minio
    spec:
      containers:
        - name: minio
          image: quay.io/minio/minio:latest
          args: ["server", "/data", "--console-address", ":9001"]
          envFrom:
            - secretRef:
                name: minio-root
          ports:
            - name: s3
              containerPort: 9000
          readinessProbe:
            httpGet:
              path: /minio/health/ready
              port: s3
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: minio-data
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: minio
spec:
  selector:
    app.kubernetes.io/name: minio
  ports:
    - name: s3
      port: 9000
      targetPort: s3
---
apiVersion: batch/v1
kind: Job
metadata:
  name: minio-make-bucket
  namespace: minio
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: mc
          image: quay.io/minio/mc:latest
          envFrom:
            - secretRef:
                name: minio-root
          command:
            - /bin/sh
            - -c
            - |
              set -eu
              mc alias set minio http://minio.minio.svc.cluster.local:9000 "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}"
              mc mb --ignore-existing minio/metoro-clickhouse-cold
```

Then point the chart at the in-cluster endpoint:

```yaml theme={null}
clickhouse:
  bundled:
    objectStorage:
      enabled: true
      provider: s3
      existingSecret:
        name: metoro-clickhouse-object-storage
      s3:
        endpoint: http://minio.minio.svc.cluster.local:9000/metoro-clickhouse-cold/clickhouse/
        region: ""
```

## Verify the Cold Tier

After installation, confirm each replica sees all three disks:

```bash theme={null}
kubectl -n metoro-hub exec chi-metoro-metoro-0-0-0 -- clickhouse-client -q \
  "SELECT name, type FROM system.disks"
```

Expected output includes `hot_pvc` (Local), `cold_object` (ObjectStorage), and `cold_cache` (ObjectStorage).

Confirm the high-volume telemetry fact tables were created on the tiered policy:

```bash theme={null}
kubectl -n metoro-hub exec chi-metoro-metoro-0-0-0 -- clickhouse-client -q \
  "SELECT name, storage_policy FROM system.tables WHERE name IN ('metoro_otel_logs_v3', 'metoro_otel_traces_v3', 'metoro_otel_metrics_v4', 'metoro_k8s_resources_v3', 'metoro_k8s_resource_liveness_v3', 'metoro_k8s_events_v2', 'metoro_otel_profiles_v2')"
```

All seven should report `hot_cold`. If they report `default`, object storage was enabled after the tables were created; see the warning above.

Other `metoro_` tables are expected to stay on the `default` policy: the `_distributed` query wrappers store no data themselves, and small side tables such as `metoro_profile_stacks`, `metoro_metric_metadata`, and `metoro_k8s_object_current` hold compact latest-state data that intentionally stays on the PVC.

Once the installation has been ingesting for longer than `hotStorageDays`, confirm parts are moving to the cold tier:

```bash theme={null}
kubectl -n metoro-hub exec chi-metoro-metoro-0-0-0 -- clickhouse-client -q \
  "SELECT disk_name, count() AS parts, sum(rows) AS rows FROM system.parts WHERE active AND table IN ('metoro_otel_logs_v3', 'metoro_otel_traces_v3', 'metoro_otel_metrics_v4', 'metoro_k8s_resources_v3', 'metoro_k8s_resource_liveness_v3', 'metoro_k8s_events_v2', 'metoro_otel_profiles_v2') GROUP BY disk_name"
```

Rows on `cold_cache` are object-backed (the cache disk fronts the object disk), and objects will be visible in the bucket under the configured prefix. Queries over old time ranges in the Metoro UI are served through the cache transparently.

## Operational Notes

* The local cache (`cacheMaxSize`) is allocated per replica on the same PVC as the hot data; include it when sizing PVCs, see [Local Disk Cache For Object Storage](/docs/on-premises/11.x/operations/clickhouse/sizing#local-disk-cache-for-object-storage).
* Growing telemetry volume grows the object store, not the PVCs, so [Upscale Hot PVC](/docs/on-premises/11.x/operations/clickhouse/upscale-hot-pvc) is only needed when ingest volume increases, not as retention accumulates.
* Object-storage credentials rotate by updating the Secret and restarting the ClickHouse pods one at a time; see [Taking Pods Offline](/docs/on-premises/11.x/operations/clickhouse/taking-pods-offline).