Object Storage Cold Tier

The bundled ClickHouse can tier telemetry between a fast hot PVC and S3-compatible object storage. Recent data stays on the hot PVC where inserts and merges are fast, and older data moves to object storage, which is cheaper and scales without PVC resizes. This is the recommended production storage model; see ClickHouse Sizing for how to size each tier. This page applies to the bundled ClickHouse only. For externally managed ClickHouse, storage tiering is owned by the team operating that service.

How It Works

When clickhouse.bundled.objectStorage is enabled, the chart configures three ClickHouse disks and a hot_cold storage policy on every replica:

Disk	Backing	Role
`hot_pvc`	The replica’s PVC	Recent data, inserts, and merges
`cold_object`	S3-compatible object storage	Long-term telemetry
`cold_cache`	Local disk cache on the PVC (`cacheMaxSize`, default 10Gi)	Caches object-backed reads

The Ingester creates every telemetry table with storage_policy = 'hot_cold' and a TTL that moves data parts to the cold volume once they are older than hotStorageDays (default 1 day). Reads are transparent: queries over older ranges are served through the cache, falling back to object storage on cache misses. Retention deletes continue to work unchanged on both tiers.

Enable object storage at install time. The storage policy and the hot-to-cold TTL are baked into each table’s schema when the Ingester first creates it. Enabling object storage on an existing installation configures the disks but leaves already-created tables on their original storage policy, so no data moves to the object store. Migrating an existing installation requires altering each table; contact Metoro support if you need to do this.

Provision the Bucket and Credentials

Provision an S3-compatible bucket (or a prefix within one) for ClickHouse before installing. The endpoint must be reachable from the hub cluster, and the credentials should be scoped to that bucket or prefix. If the endpoint uses a private CA, include it in the chart’s trustedCAs material. Create the credentials Secret in the hub namespace:

kubectl -n metoro-hub create secret generic metoro-clickhouse-object-storage \
  --from-literal=access_key_id="CHANGE_ME_ACCESS_KEY_ID" \
  --from-literal=secret_access_key="CHANGE_ME_SECRET_ACCESS_KEY"

Configure Helm Values

clickhouse:
  bundled:
    objectStorage:
      enabled: true
      provider: s3
      existingSecret:
        name: metoro-clickhouse-object-storage
      s3:
        # Full URL including the bucket and an optional key prefix, with a
        # trailing slash. All replicas can share one prefix.
        endpoint: https://s3.eu-west-2.amazonaws.com/metoro-clickhouse-cold/clickhouse/
        region: eu-west-2
      # Days of data kept on the hot PVC before parts move to object storage.
      hotStorageDays: 1
      # Per-replica local cache in front of the object disk.
      cacheMaxSize: 10Gi

The endpoint is passed to ClickHouse’s S3 disk as-is, so it must contain the bucket name and any prefix. Path-style URLs (https://host:port/bucket/prefix/) work for MinIO and most self-hosted S3 implementations; virtual-hosted style (https://bucket.s3.region.amazonaws.com/prefix/) works for AWS. Leave region empty for implementations that do not use it. Azure Blob Storage is also supported with provider: azureBlob; it reads account_name and account_key from the same Secret and takes azureBlob.storageAccountUrl and azureBlob.containerName instead of the s3 block. On startup, each ClickHouse replica verifies it can write to the object store and fails loudly if the endpoint or credentials are wrong, so misconfiguration surfaces immediately rather than at the first TTL move.

Example: In-Cluster MinIO

For evaluation clusters, or air-gapped environments without an existing object store, an in-cluster MinIO provides an S3-compatible endpoint that exercises the exact same provider: s3 path. The single-replica setup below is suitable for testing; for production MinIO, follow MinIO’s own distributed-deployment guidance. Create the namespace and root credentials, and give ClickHouse the same credentials:

kubectl create namespace minio

MINIO_USER="metoro-clickhouse"
MINIO_PASS="$(openssl rand -hex 24)"

kubectl -n minio create secret generic minio-root \
  --from-literal=MINIO_ROOT_USER="${MINIO_USER}" \
  --from-literal=MINIO_ROOT_PASSWORD="${MINIO_PASS}"

kubectl -n metoro-hub create secret generic metoro-clickhouse-object-storage \
  --from-literal=access_key_id="${MINIO_USER}" \
  --from-literal=secret_access_key="${MINIO_PASS}"

Deploy MinIO with a persistent volume, a Service, and a one-shot Job that creates the bucket:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-data
  namespace: minio
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 128Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: minio
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: minio
  template:
    metadata:
      labels:
        app.kubernetes.io/name: minio
    spec:
      containers:
        - name: minio
          image: quay.io/minio/minio:latest
          args: ["server", "/data", "--console-address", ":9001"]
          envFrom:
            - secretRef:
                name: minio-root
          ports:
            - name: s3
              containerPort: 9000
          readinessProbe:
            httpGet:
              path: /minio/health/ready
              port: s3
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: minio-data
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: minio
spec:
  selector:
    app.kubernetes.io/name: minio
  ports:
    - name: s3
      port: 9000
      targetPort: s3
---
apiVersion: batch/v1
kind: Job
metadata:
  name: minio-make-bucket
  namespace: minio
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: mc
          image: quay.io/minio/mc:latest
          envFrom:
            - secretRef:
                name: minio-root
          command:
            - /bin/sh
            - -c
            - |
              set -eu
              mc alias set minio http://minio.minio.svc.cluster.local:9000 "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}"
              mc mb --ignore-existing minio/metoro-clickhouse-cold

Then point the chart at the in-cluster endpoint:

clickhouse:
  bundled:
    objectStorage:
      enabled: true
      provider: s3
      existingSecret:
        name: metoro-clickhouse-object-storage
      s3:
        endpoint: http://minio.minio.svc.cluster.local:9000/metoro-clickhouse-cold/clickhouse/
        region: ""

Verify the Cold Tier

After installation, confirm each replica sees all three disks:

kubectl -n metoro-hub exec chi-metoro-metoro-0-0-0 -- clickhouse-client -q \
  "SELECT name, type FROM system.disks"

Expected output includes hot_pvc (Local), cold_object (ObjectStorage), and cold_cache (ObjectStorage). Confirm the high-volume telemetry fact tables were created on the tiered policy:

kubectl -n metoro-hub exec chi-metoro-metoro-0-0-0 -- clickhouse-client -q \
  "SELECT name, storage_policy FROM system.tables WHERE name IN ('metoro_otel_logs_v3', 'metoro_otel_traces_v3', 'metoro_otel_metrics_v4', 'metoro_k8s_resources_v3', 'metoro_k8s_resource_liveness_v3', 'metoro_k8s_events_v2', 'metoro_otel_profiles_v2')"

All seven should report hot_cold. If they report default, object storage was enabled after the tables were created; see the warning above. Other metoro_ tables are expected to stay on the default policy: the _distributed query wrappers store no data themselves, and small side tables such as metoro_profile_stacks, metoro_metric_metadata, and metoro_k8s_object_current hold compact latest-state data that intentionally stays on the PVC. Once the installation has been ingesting for longer than hotStorageDays, confirm parts are moving to the cold tier:

kubectl -n metoro-hub exec chi-metoro-metoro-0-0-0 -- clickhouse-client -q \
  "SELECT disk_name, count() AS parts, sum(rows) AS rows FROM system.parts WHERE active AND table IN ('metoro_otel_logs_v3', 'metoro_otel_traces_v3', 'metoro_otel_metrics_v4', 'metoro_k8s_resources_v3', 'metoro_k8s_resource_liveness_v3', 'metoro_k8s_events_v2', 'metoro_otel_profiles_v2') GROUP BY disk_name"

Rows on cold_cache are object-backed (the cache disk fronts the object disk), and objects will be visible in the bucket under the configured prefix. Queries over old time ranges in the Metoro UI are served through the cache transparently.

Operational Notes

The local cache (cacheMaxSize) is allocated per replica on the same PVC as the hot data; include it when sizing PVCs, see Local Disk Cache For Object Storage.
Growing telemetry volume grows the object store, not the PVCs, so Upscale Hot PVC is only needed when ingest volume increases, not as retention accumulates.
Object-storage credentials rotate by updating the Secret and restarting the ClickHouse pods one at a time; see Taking Pods Offline.

Get Started

AI SRE

Concepts

Ingestion Settings

Traces

Logs

Metrics

Profiling

Kubernetes Resources

Dashboards

Infrastructure

Advisor

Alerts & Monitoring

Integrations

Uptime Monitoring

User Management

On-Premises

Administration

How It Works

Provision the Bucket and Credentials

Configure Helm Values

Example: In-Cluster MinIO

Verify the Cold Tier

Operational Notes

​How It Works

​Provision the Bucket and Credentials

​Configure Helm Values

​Example: In-Cluster MinIO

​Verify the Cold Tier

​Operational Notes

How It Works

Provision the Bucket and Credentials

Configure Helm Values

Example: In-Cluster MinIO

Verify the Cold Tier

Operational Notes