On-Prem

On-Premises Installation & Management

This guide provides detailed instructions for installing and managing Metoro in an on-premises environment. It covers system requirements, installation steps, and best practices for maintaining your deployment.

Prerequisites

Before beginning the installation, ensure your environment meets the following requirements:

  • Kubernetes cluster (v1.19 or later)
  • Helm 3.x installed
  • Resource requirements per node for the Metoro Agent:
    • CPU: 0.3 cores
    • Memory: 300MB RAM
  • Total resource requirements for the Metoro Hub:
    • CPU: 4 cores
    • Memory: 8GB RAM
  • Network requirements:
    • Access to quay.io/metoro repositories for pulling images (optional if using your own private registry)
    • Internal network connectivity between cluster nodes
    • Ingress controller for external access (recommended)

Quick Start

1. Get Access to Required Resources

Contact us to get access to the Helm charts and private image repositories:

You will receive:

  • Helm repository (zipped)
  • Image repository pull secret

2. Prepare the Installation

  1. Extract the helm chart:
unzip helm.zip && cd helm
  1. Set your kubectl context:
kubectl config use-context CLUSTER_YOU_WANT_TO_INSTALL_INTO

3. Install Metoro Hub

Install the Metoro hub using Helm:

helm upgrade --install \
  --namespace metoro-hub \
  --create-namespace \
  metoro ./ \
  --set clickhouse.enabled=true \
  --set postgresql.enabled=true \
  --set onPrem.isOnPrem=true \
  --set imagePullSecret.data=<imagePullSecret-from-step-1> \
  --set apiserver.replicas=1 \
  --set ingester.replicas=1 \
  --set temporal.enabled=true \
  --set ingester.autoscaling.horizontalPodAutoscaler.enabled=false \
  --set apiserver.autoscaling.horizontalPodAutoscaler.enabled=false

Note

If the Clickhouse pod remains in pending state, it's likely due to insufficient cluster resources. You can adjust the resource limits in the Clickhouse StatefulSet definition.

4. Access the UI

  1. Port forward the API server:
kubectl port-forward -n metoro-hub service/apiserver 8080:80
  1. Create an account:

5. Install the Metoro Agent

  1. After logging in, select "Existing Cluster" and enter your cluster's name
  2. Copy the exporter.secret.bearerToken value from the installation screen
  3. Run the installation command:
bash -c "$(curl -fsSL http://localhost:8080/install.sh)" -- \
  TOKEN_HERE \
  http://ingester.metoro-hub.svc.cluster.local/ingest/api/v1/otel \
  http://apiserver.metoro-hub.svc.cluster.local/api/v1/exporter \
  --existing-cluster \
  --on-prem

Advanced Configuration - Production

Minimal Production Configuration

For the metoro-hub values.yaml

clickhouse:
  enabled: true
  auth:
    password: "CHANGE_ME_CLICKHOUSE_PASSWORD" # Use a random password

postgresql:
  enabled: true
  auth:
    password: "CHANGE_ME_POSTGRES_PASSWORD" # Use a random password

onPrem:
  isOnPrem: true

imagePullSecret:
  data: "IMAGE_PULL_SECRET"

authSecret:
  authMaterial: "CHANGE_ME_AUTH_MATERIAL" # Use a random string

apiserver:
  replicas: 2
  autoscaling:
    horizontalPodAutoscaler:
      enabled: false
  defaultOnPremAdmin:
    email: "YOUR_EMAIL_CHANGE_ME"
    password: "YOUR_PASSWORD_CHANGE_ME"
    name: "YOUR NAME_CHANGE_ME"
    organization: "YOUR_ORGANIZATION_CHANGE_ME"
    environmentName: "YOUR_ENVIRONMENT_NAME_CHANGE_ME"


temporal:
  enabled: true
  server:
    config:
      persistence:
        default:
          sql:
            password: "CHANGE_ME_POSTGRES_PASSWORD" # Use the same password as the postgres above
        visibility:
          sql:
            password: "CHANGE_ME_POSTGRES_PASSWORD" # Use the same password as the postgres above

ingester:
  replicas: 2
  autoscaling:
    horizontalPodAutoscaler:
      enabled: false

Then install with the following command:

helm upgrade --install --namespace metoro-hub --create-namespace metoro ./ -f values.yaml

For the metoro-exporter values.yaml:

exporter:
  image:
    tag: "0.841.0"
  envVars:
    mandatory:
      otlpUrl: "http://ingester.metoro-hub.svc.cluster.local/ingest/api/v1/otel"
      apiServerUrl: "http://apiserver.metoro-hub.svc.cluster.local/api/v1/exporter"
  secret:
    externalSecret:
      enabled: true
      name: "on-prem-default-exporter-token-secret"
      secretKey: "token"

nodeAgent:
  image:
    tag: "0.65.0"

Then install with the following command:

helm repo add metoro-exporter https://metoro-io.github.io/metoro-helm-charts/ ;
helm repo update metoro-exporter;
helm upgrade --install --create-namespace --namespace metoro metoro-exporter metoro-exporter/metoro-exporter -f values.yaml

Securing the Metoro Hub

Before deploying in production, you should change at least the following settings in the Metoro Hub Helm chart:

apiserver:
  defaultOnPremAdmin:
    password: "CHANGE_ME_TO_SECURE_PASSWORD"  # Change this to a secure password, you'll use this to log in to the UI for the first time

postgresql:
  auth:
    password: "CHANGE_ME_POSTGRES_PASSWORD" # Use a random password

clickhouse:
  auth:
    password: "CHANGE_ME_CLICKHOUSE_PASSWORD" # Use a random password

authSecret:
  authMaterial: "CHANGE_ME_AUTH_MATERIAL" # Use a random string

temporal:
  server:
    config:
      persistence:
        default:
          sql:
            password: "CHANGE_ME_POSTGRES_PASSWORD" # Use the same password as above
        visibility:
          sql:
            password: "CHANGE_ME_POSTGRES_PASSWORD" # Use the same password as above

onPrem:
  isOnPrem: true

Connecting the exporter to the Metoro Hub via helm

The exporter needs to be configured to connect to the Metoro hub. This can either be done through the UI or by setting the following values in the hub helm chart:

apiserver:
  defaultOnPremAdmin:
    email: "YOUR_EMAIL"
    password: "YOUR_PASSWORD"
    name: "YOUR NAME"
    organization: "YOUR ORGANIZATION"
    environmentName: "YOUR ENVIRONMENT NAME"

Then when installing the exporter, you can set the following values:

exporter:
  secret:
    externalSecret:
      enabled: true
      name: "on-prem-default-exporter-token-secret"
      secretKey: "token"

Using a different image registry

If you want to use a different image registry, you can set the imagePullSecret field in the Helm chart values file to a secret containing the pull secret.

imagePullSecret:
  name: "my-registry-credentials"
  data: "dockerconfigjson-encoded-value"

High Availability Setup

For production environments requiring high availability. We also recommend using external databases for increased availability and performance. Check out the external database configuration section for more details. The postgres chart doesn't have great support for HA. The Clickhouse chart has built-in HA support.

ingester:
  replicas: 2
  autoscaling:
    horizontalPodAutoscaler:
      enabled: true
      minReplicas: 2
      maxReplicas: 4
      targetCPUUtilizationPercentage: 60

apiserver:
  replicas: 2
  autoscaling:
    horizontalPodAutoscaler:
      enabled: true
      minReplicas: 2
      maxReplicas: 4
      targetCPUUtilizationPercentage: 60

clickhouse:
  enabled: true
  persistence:
    size: 100Gi
  replicaCount: 3

postgresql:
  enabled: true
  persistence:
    size: 20Gi
  primary:
    replicaCount: 3

External Database Configuration

To use external databases instead of the built-in ones:

clickhouse:
  enabled: false

clickhouseSecret:
  name: "clickhouse-secret"
  clickhouseUrl: "clickhouse://xxxxxxx.us-east-1.aws.clickhouse.cloud:9440"
  clickhouseUser: "username"
  clickhousePassword: "password"
  clickhouseDatabase: "metoro"

postgresql:
  enabled: false

postgresSecret:
  name: "postgres-secret"
  postgresHost: "prod-us-east.cluster-xxxxxxx.us-east-1.rds.amazonaws.com"
  postgresPort: "5432"
  postgresUser: "postgres"
  postgresPassword: "password"
  postgresDatabase: "metoro"

# This needs to be matched with the postgresSecret values
temporal:
  server:
    config:
      persistence:
        default:
          driver: sql
          sql:
            driver: postgres12
            database: temporal
            user: postgres
            password: password
            host: "prod-us-east.cluster-xxxxxxx.us-east-1.rds.amazonaws.com"
            port: 5432
        visibility:
          driver: sql
          sql:
            driver: postgres12
            database: temporal_visibility
            user: postgres
            password: CHANGE_ME
            host: "prod-us-east.cluster-xxxxxxx.us-east-1.rds.amazonaws.com"
            port: 5432

Ingress Configuration

Enable ingress for external access:

apiserver:
  # Match this with the hostname of the ingress
  deploymentUrl: http(s)://metoro.yourdomain.com
  ingress:
    enabled: true
    className: "nginx"
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    hosts:
      - host: "metoro.yourdomain.com"
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: metoro-tls
        hosts:
          - metoro.yourdomain.com

ingester:
  ingress:
    enabled: true
    className: "nginx"
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    hosts:
      - host: "ingest.metoro.yourdomain.com"
        paths:
          - path: /
            pathType: Prefix
    tls:
      - secretName: metoro-ingester-tls
        hosts:
          - ingest.metoro.yourdomain.com

Maintenance

Upgrading Metoro

Minor version upgrades can just be installed using a helm upgrade command.

helm upgrade --install --namespace metoro-hub ./ -f values.yaml

Major version upgrades will require a more in-depth migration process. Each major release will have a migration guide available on the Metoro website and in the helm chart itself.

Support and Resources

For additional support:

Full configuration reference

Below is the full list of configuration options for the Metoro Helm chart.

KeyTypeDefaultDescription
apiserver.defaultOnPremAdmin.emailstring"admin@metoro.io"Default admin email address set up on first login
apiserver.defaultOnPremAdmin.environmentNamestring"Default Environment"Default environment name set up on first login
apiserver.defaultOnPremAdmin.namestring"Admin"Default admin name set up on first login
apiserver.defaultOnPremAdmin.organizationstring"Default Organization"Default organization name set up on first login
apiserver.defaultOnPremAdmin.passwordstring"admin123"Default admin password set up on first login
apiserver.defaultOnPremAdmin.serviceAccount.annotationsobject{}Service account annotations
apiserver.defaultOnPremAdmin.serviceAccount.createbooleantrueWhether to create a service account
apiserver.deploymentUrlstring"https://somedeploymenturl.tld..."Deployment URL for the API server
apiserver.image.pullPolicystring"IfNotPresent"Image pull policy for API server container
apiserver.image.repositorystring"quay.io/metoro/metoro-apiserver"Docker image repository for API server
apiserver.ingress.annotationsobject{"kubernetes.io/ingress.class": "nginx"}Ingress annotations for API server
apiserver.ingress.classNamestring"nginx"Ingress class name for API server
apiserver.ingress.enabledbooleanfalseEnable/disable ingress for API server
apiserver.ingress.hosts[0].hoststring"api.local.test"Ingress hostname for API server
apiserver.ingress.hosts[0].paths[0].pathstring"/"Path for ingress rule
apiserver.ingress.hosts[0].paths[0].pathTypestring"Prefix"Path type for ingress rule
apiserver.namestring"apiserver"Name of the API server component
apiserver.replicasinteger4Number of API server replicas
apiserver.resources.limits.cpustring/number4CPU resource limit for API server
apiserver.resources.limits.memorystring"16Gi"Memory resource limit for API server
apiserver.resources.requests.cpustring/number1Requested CPU resources for API server
apiserver.resources.requests.memorystring"2Gi"Requested memory resources for API server
apiserver.service.namestring"apiserver"Name of the API server service
apiserver.service.portinteger80Service port for API server
apiserver.service.targetPortinteger8080Target port for API server service
apiserver.service.typestring"ClusterIP"Kubernetes service type for API server
authSecret.authMaterialstring"SOME_AUTH_MATERIAL"Authentication material used to sign JWTs for API server and ingester
authSecret.namestring"auth-secret"Name of the authentication secret
awsSecret.credentialsFileContentsstring"SOME_AWS_SECRET"AWS credentials file contents, currently not support in on-premises
awsSecret.namestring"aws-secret"Name of the AWS secret, currently not support in on-premises
awsSesSecret.awsSesAccessKeyIdstring"SOME_AWS_SES_ACCESS_KEY_ID"AWS SES access key ID, currently not support in on-premises
awsSesSecret.awsSesRegionstring"SOME_AWS_SES_REGION"AWS SES region, currently not support in on-premises
awsSesSecret.awsSesSecretKeystring"SOME_AWS_SES_SECRET_KEY"AWS SES secret key, currently not support in on-premises
awsSesSecret.namestring"aws-ses-secret"Name of the AWS SES secret, currently not support in on-premises
clickhouse.containerPorts.tcpinteger9440TCP container port for in-cluster ClickHouse
clickhouse.containerPorts.tcpSecureinteger20434Secure TCP container port for in-cluster ClickHouse
clickhouse.enabledbooleanfalseEnable/disable in-cluster ClickHouse installation
clickhouse.persistence.sizestring"100Gi"Storage size for in-cluster ClickHouse
clickhouse.replicaCountinteger1Number of ClickHouse replicas
clickhouse.resourcesPresetstring"2xlarge"Resource preset for ClickHouse
clickhouse.secret.clickhouseDatabasestring"SOME_CLICKHOUSE_DATABASE"ClickHouse database name
clickhouse.secret.clickhousePasswordstring"SOME_CLICKHOUSE_PASSWORD"ClickHouse password
clickhouse.secret.clickhouseUrlstring"SOME_CLICKHOUSE_HOST"ClickHouse URL
clickhouse.secret.clickhouseUserstring"SOME_CLICKHOUSE_USER"ClickHouse user
clickhouse.secret.namestring"clickhouse-secret"Name of the ClickHouse secret
clickhouse.service.ports.tcpinteger9440TCP port for ClickHouse
clickhouse.service.ports.tcpSecureinteger20434Secure TCP port for ClickHouse
clickhouse.shardsinteger1Number of ClickHouse shards
clickhouse.zookeeper.enabledbooleanfalseEnable/disable ZooKeeper for ClickHouse
environmentstring"none"Environment for the deployment, e.g. "dev", "test", "prod"
imagePullSecret.datastring"SOME_DOCKERHUB_CREDENTIAL"Registry credentials in dockerconfigjson format
imagePullSecret.namestring"dockerhub-credentials"Name of the Docker registry credentials secret
ingester.autoscaling.horizontalPodAutoscaler.enabledbooleantrueEnable/disable HPA for ingester
ingester.autoscaling.horizontalPodAutoscaler.maxReplicasinteger10Maximum number of replicas for HPA
ingester.autoscaling.horizontalPodAutoscaler.minReplicasinteger4Minimum number of replicas for HPA
ingester.autoscaling.horizontalPodAutoscaler.namestring"metoro-ingester-hpa"Name of the HPA
ingester.autoscaling.horizontalPodAutoscaler.targetCPUUtilizationPercentageinteger60Target CPU utilization percentage for scaling
ingester.configMap.namestring"ingester-config"Name of the ingester ConfigMap
ingester.image.pullPolicystring"IfNotPresent"Image pull policy for ingester container
ingester.image.repositorystring"quay.io/metoro/metoro-ingester"Docker image repository for ingester
ingester.ingress.annotationsobject{"kubernetes.io/ingress.class": "nginx"}Ingress annotations
ingester.ingress.classNamestring"nginx"Ingress class name
ingester.ingress.enabledbooleanfalseEnable/disable ingress for ingester
ingester.ingress.hosts[0].hoststring"ingester.local.test"Ingress hostname
ingester.ingress.hosts[0].paths[0].pathstring"/"Path for ingress rule
ingester.ingress.hosts[0].paths[0].pathTypestring"Prefix"Path type for ingress rule
ingester.namestring"ingester"Name of the ingester component
ingester.replicasinteger4Number of ingester replicas to deploy
ingester.resources.limits.cpustring/number4CPU resource limit for each ingester pod
ingester.resources.limits.memorystring"16Gi"Memory resource limit for each ingester pod
ingester.resources.requests.cpustring/number1Requested CPU resources for each ingester pod
ingester.resources.requests.memorystring"2Gi"Requested memory resources for each ingester pod
ingester.service.namestring"ingester"Name of the ingester service
ingester.service.portinteger80Service port for ingester
ingester.service.targetPortinteger8080Target port for ingester service
ingester.service.typestring"ClusterIP"Kubernetes service type for ingester
klaviyoSecret.klaviyoApiKeystring"SOME_KLAVIYO_API_KEY"Klaviyo API key, currently not support in on-premises
klaviyoSecret.namestring"klaviyo-secret"Name of the Klaviyo secret, currently not support in on-premises
onPrem.isOnPrembooleanfalseFlag for on-premises deployment, should always be set to true on-premises
pagerDutySecret.namestring"pagerduty-secret"Name of the PagerDuty secret, currently not support in on-premises
pagerDutySecret.pagerDutyClientIdstring"SOME_PAGERDUTY_CLIENT_ID"PagerDuty client ID, currently not support in on-premises
pagerDutySecret.pagerDutyClientSecretstring"SOME_PAGERDUTY_CLIENT_SECRET"PagerDuty client secret, currently not support in on-premises
postgresSecret.namestring"postgres-secret"Name of the PostgreSQL secret for external database
postgresSecret.postgresDatabasestring"SOME_POSTGRES_DATABASE"PostgreSQL database name for external database
postgresSecret.postgresHoststring"SOME_POSTGRES_HOST"PostgreSQL host for external database
postgresSecret.postgresPasswordstring"SOME_POSTGRES_PASSWORD"PostgreSQL password for external database
postgresSecret.postgresPortstring"SOME_POSTGRES_PORT"PostgreSQL port for external database
postgresSecret.postgresUserstring"SOME_POSTGRES_USER"PostgreSQL user for external database
postgresql.enabledbooleanfalseEnable/disable in-cluster PostgreSQL installation
postgresql.auth.postgresPasswordstring"CHANGE_ME"PostgreSQL password for in-cluster database
postgresql.persistence.sizestring"2Gi"Storage size for in-cluster PostgreSQL
slackSecret.namestring"slack-secret"Name of the Slack secret, currently not support in on-premises
slackSecret.slackClientIdstring"SOME_SLACK_CLIENT_ID"Slack client ID, currently not support in on-premises
slackSecret.slackClientSecretstring"SOME_SLACK_CLIENT_SECRET"Slack client secret, currently not support in on-premises
stripeSecret.namestring"stripe-secret"Name of the Stripe secret, currently not support in on-premises
stripeSecret.stripeKeystring"SOME_STRIPE_KEY"Stripe API key, currently not support in on-premises
temporal.admintools.image.tagstring"1.24.2-tctl-1.18.1-cli-0.13.2"Temporal admin tools image tag
temporal.cassandra.enabledbooleanfalseEnable/disable Cassandra for Temporal, currently not support in on-premises
temporal.elasticsearch.enabledbooleanfalseEnable/disable Elasticsearch for Temporal, currently not support in on-premises
temporal.enabledbooleanfalseEnable/disable Temporal, should be set to true in production
temporal.grafana.enabledbooleanfalseEnable/disable Grafana for Temporal, currently not support in on-premises
temporal.mysql.enabledbooleanfalseEnable/disable MySQL for Temporal, currently not support in on-premises
temporal.postgres.enabledbooleantrueEnable/disable PostgreSQL for Temporal, should be set to true in production
temporal.prometheus.enabledbooleanfalseEnable/disable Prometheus for Temporal, currently not support in on-premises
temporal.schema.setup.enabledbooleantrueEnable/disable Temporal schema setup, should be set to true in production
temporal.schema.update.enabledbooleantrueEnable/disable Temporal schema updates, should be set to true in production
temporal.server.replicaCountinteger1Number of Temporal server replicas, should be set to 1 in production, 2 or more for HA
temporalEndpointstring"temporal-frontend:7233"Temporal service endpoint
versions.dev.apiserverstring"0.856.0"API server version for dev
versions.dev.isdevbooleanfalseDevelopment version flag
versions.dev.ingesterstring"0.856.0"Ingester version for dev
versions.onprem.apiserverstring"0.856.0"API server version for on-prem
versions.onprem.isonprembooleantrueOn-premises version flag
versions.onprem.ingesterstring"0.856.0"Ingester version for on-prem
versions.prod.apiserverstring"0.856.0"API server version for prod
versions.prod.isprodbooleanfalseProduction version flag
versions.prod.ingesterstring"0.856.0"Ingester version for prod
Previous
Kubernetes Resources