Skip to main content
To add your webhook to Metoro, go to the integrations page and click on the Add Webhook button. Fill in the details of the webhook and click on the Add Webhook button. Add Webhook Integration modal with fields for name, URL, HTTP method, headers, and body template
  • Name (required): The name of the webhook
  • URL (required): The URL of the webhook. You can use template variables in the URL for dynamic values (see template variables section below)
  • HTTP Method: The HTTP method to use when sending the webhook. Default is POST
  • Headers: Additional headers to send with the webhook. If you are sending a POST request, Content-Type header is added by default with the value application/json. You can add more headers by clicking on the Add Header button.
  • Body Template (optional): If you would like to send a custom body with the webhook, you can use the body template. Leave empty to use the default JSON payload. You can use template variables in your webhook body (see template variables section below).
  • Folder Location: The filesystem folder where the webhook is saved. Manual webhooks default to /webhooks/default/, and custom team folders must live under /webhooks/.
Webhook folders use the same folder-level permissions as alerts and dashboards. Users only see webhook integrations whose file path they can read, and creating, editing, moving, or deleting a webhook checks permissions against that webhook’s filesystem path.

Kubernetes-managed webhooks

Webhook integrations can be declared as Kubernetes custom resources with MetoroWebhook under the observability.metoro.io/v1alpha1 API group, so the destinations your alerts notify live in version control and roll out through GitOps alongside the alerts that use them. Credentials never need to appear in the manifest: headersFrom and bodyFrom reference Kubernetes Secrets in the webhook’s own namespace, and Metoro resolves them only at the moment a webhook is delivered.
Webhooks synced from Kubernetes appear on the integrations page with a Managed via K8s badge and are read-only in the UI: Kubernetes is the source of truth, so edit or delete the custom resource instead. Each synced webhook gets the deterministic integration ID kubernetes-managed.<cluster>.<namespace>.<name> — which is how alert manifests reference it — and lands in the filesystem at /webhooks/kubernetes-managed/<cluster>/<namespace>/, where the usual folder permissions apply.

How it works

  • The MetoroWebhook CRD is installed by the Metoro exporter chart on every monitored cluster, so it is available on both Metoro cloud and on-prem. The base resource ships with metoro-exporter chart version 0.476.0 or later (Metoro cloud) and metoro-exporter-onprem 11.0.0 or later; the headersFrom/bodyFrom Secret references require metoro-exporter 0.477.0 or later (or the latest 11.x on-prem exporter chart). On an older chart the fields are silently dropped by the Kubernetes API server, so upgrade the exporter chart before using them.
  • Most MetoroWebhook creates, spec updates, and deletions appear in Metoro in about 30 seconds; allow up to one minute for the exporter and sync workflow to process a change.
  • If the exporter is configured with a METORO_K8S_RESOURCES allowlist restricting which resources it watches, the list must include metoro-webhook (or another MetoroWebhook alias); otherwise applied webhooks are never discovered.
  • List synced webhooks with kubectl get metorowebhooks -A (short names: mw, mwebhook).

Referencing Kubernetes Secrets

Inline headers and body values are stored in the Kubernetes resource and in Metoro’s exported resource history. To keep credentials out of both, reference a Secret with headersFrom (per header) or bodyFrom (for the whole body template):
  • References always resolve in the webhook’s own namespace. A secretKeyRef carries only name and key; cross-namespace references are not representable.
  • Values are resolved at delivery time. The Metoro exporter mirrors just the referenced Secret keys to an encrypted store; the CRD, its exported snapshot, the integration record, the UI, and delivery failure messages never contain the values.
  • Rotation needs no CRD change. The exporter re-syncs referenced Secrets every 60 seconds, so an edited Secret takes effect on the next delivery within about a minute.
  • Deleting a Secret revokes it. The mirrored values are removed on the next sync and deliveries fail (rather than sending stale credentials) until the Secret is recreated.

Enabling Secret syncing

Secret syncing is opt-in on the exporter because it grants get access to Secrets:
By default access is scoped to the exporter’s release namespace through a namespaced, get-only Role — the exporter cannot list or enumerate Secrets in any mode, and cannot read outside the configured namespaces regardless of what a manifest references. Webhooks in other namespaces need their namespace added to namespaces (or global: true) before their references resolve.
Kubernetes RBAC is the trust boundary. Anyone who can create a MetoroWebhook with a secretKeyRef in an in-scope namespace can have the named Secret’s referenced keys used in webhook deliveries, so grant create on metorowebhooks accordingly.
On Metoro cloud the encrypted store is fully managed and there is nothing else to configure. On-prem installations must configure the hub apiserver with an encryption key before references work: create a Kubernetes Secret in the hub namespace holding a base64-encoded 32-byte key (openssl rand -base64 32) and point apiserver.secretStore.encryptionSecret.name/.key at it in the hub chart values. Never regenerate this key once set — mirrored values are sealed with it and become permanently undecryptable under a new key.

Spec reference

A header name may not appear in both headers and headersFrom (compared case-insensitively), and Content-Type must always be set inline in headers.

End-to-end example

A complete, appliable test: a Secret carrying the credentials, a webhook referencing them, and an alert that fires immediately so you can watch the delivery arrive. Point the URL at a webhook.site inbox to inspect the result.
Replace <cluster> in the alert’s webhookDestination.uuid with your cluster’s environment name — the alert references the webhook by its deterministic ID, so it works whether the webhook has synced yet or not. Within a couple of minutes of applying, the alert fires and webhook.site shows the delivered request:
The Authorization header carries the Secret’s value — resolved at delivery, never stored in the manifest — and the body is the bodyFrom template with $alert_name and $alert_state substituted. Delete the alert when you’re done: it stays firing by design.

Validation and failure modes

  • A MetoroWebhook that passes the CRD schema but fails sync-time validation (a header in both headers and headersFrom, Content-Type in headersFrom, both body and bodyFrom set, or a malformed secretKeyRef) is skipped and does not create or update an integration. The reason is written into your own logs as an error record in the metoro-internal environment under the service metoro-webhook-sync, so you can find and alert on it.
  • If a reference cannot be resolved at delivery time, the webhook is not sent — a request delivered without its Authorization header is worse than one not delivered. The delivery failure names the Secret and a reason (missing_secret, missing_key, …) but never a value.
  • missing_secret/missing_key are retried across the exporter’s sync interval, which covers the window where a freshly applied webhook fires before its Secret has been mirrored (typically the webhook’s first minute of existence).
  • If Secret syncing is not enabled on the exporter (or the namespace is out of scope), references never resolve and deliveries fail with missing_secret; enable exporter.secretRefs as above.

Exporting an existing webhook

Any webhook created in the UI can be exported as a kubectl-appliable MetoroWebhook manifest with the download icon on the integrations page — useful for moving webhooks into GitOps-managed configuration. The export includes any Secret references, with a comment naming the Secrets the manifest expects in its target namespace. Applying the manifest creates a new Kubernetes-managed webhook with the deterministic ID; update alerts to reference it and delete the original to avoid duplicate notifications.

Template Variables

You can use template variables in both the webhook URL and body template. The available variables depend on what triggered the webhook: an alert or an AI SRE (Guardian) notification. The same webhook integration can be used as a destination for both.

Alert variables

  • $alert_name: The name of the alert
  • $alert_description: The description of the alert
  • $alert_uuid: The UUID of the alert definition
  • $alert_fire_uuid: The UUID of the specific alert fire instance
  • $alert_state: The state of the alert (either “firing” or “resolved”)
  • $deep_link: Direct link to view the alert details in Metoro
  • $environment: [Deprecated - use $attributes instead] Environment context of the alert. It’s set for Kubernetes and Log alerts only or if the alert has a group by with environment
  • $service: [Deprecated - use $attributes instead] The service associated with the alert. It’s set for Kubernetes and Log alerts only or if the alert has a group by with service.name/service_name/client.service.name/server.service.name.
  • $fired_at: Unix Timestamp when alert was fired
  • $resolved_at: Unix Timestamp when alert was resolved/recovered.
  • $breaching_datapoint_value: The last metric/trace value that triggered the alert. It’s set for Trace and Metric alerts only.
  • $breaching_datapoint_time: Unix Timestamp of the last breaching value. It’s set for Trace and Metric alerts only.
  • $metric_name: Name of the metric that triggered the alert. It’s set for Trace and Metric alerts only.
  • $attributes: List of attribute key and value pairs (in key: value format) for which the alert is firing. Only set when you use a group by in your alert.

AI SRE (Guardian) notification variables

Webhooks can also be used as destinations for AI SRE notifications (deployment monitoring, autonomous investigations, issue notifications, and agent action approvals). These notifications provide the following variables:
  • $title: The notification title, as plain text
  • $message: The notification message, as plain text
  • $type: The specific event type (see the table below) — use this to distinguish notifications on your end
  • $deep_link: Direct link to view the relevant page in Metoro (may be empty for some notification types where the link is embedded in the message instead)
Each notification also exposes its attributes as individual template variables (e.g. $service, $investigationUUID). The built-in variables above are reserved and always win: issue and investigation notifications carry a title attribute, but $title resolves to the full notification title, so that attribute is not accessible as a template variable and is not listed below. The available event types and template variables:

Default Guardian webhook payload

If no body template is set, Guardian notifications send a structured JSON event:
  • id is the UUID of the underlying event — stable across delivery retries, use it for deduplication.
  • type determines the shape of the typed data object; the full schema for every event type is in the Webhook Events section of the API reference (starting at deployment rollout detected).

Examples

URL Examples

You can include template variables directly in your webhook URL:

Body Template Examples

  1. JSON format:
  1. Plain text format:
  1. XML format:
Remember to set the Content-Type header in the headers section with the appropriate value.When the body is sent as JSON (an explicit Content-Type containing json, or the default application/json on POST/PUT requests), substituted values are automatically JSON-string-escaped so multiline messages and quotes don’t break your template — write "message": "$message" without worrying about escaping. For any other content type, values are substituted verbatim.
After your webhook is added, you can select your webhook as a destination in the alert creation wizard. Alert destination selector with Webhook option selected and webhook dropdown showing available webhooks
You can check webhook delivery status and errors in Metoro Logs view. This provides visibility into successful deliveries and any issues encountered. Go to the Logs page and filter by environment=metoro-internal and service.name=metoro-webhooks to see webhook notifications that are successfully sent or failed with errors.