> ## Documentation Index
> Fetch the complete documentation index at: https://metoro.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Alert Destinations

> Configure where and how your alerts are delivered, including recovery notifications

<Info>
  **New Feature**: Recovery notifications are now available for webhook, Slack, and email destinations! Get notified when your alerts return to normal state.
</Info>

Alert destinations determine where notifications are sent when alerts are triggered or resolved. Metoro supports multiple destination types, allowing you to route alerts to the right teams and systems.

## Supported Destination Types

<CardGroup cols={2}>
  <Card title="Slack" icon="slack" color="#4A154B">
    Send alert notifications directly to Slack channels for immediate team visibility.

    **Key Features:**

    * Direct channel integration
    * Rich message formatting
    * Alert status and details
    * Recovery notifications support
    * Deep links to Metoro
  </Card>

  <Card title="PagerDuty" icon="pager" color="#06AC38">
    Integrate with PagerDuty for on-call management and incident response.

    **Key Features:**

    * Automatic incident creation
    * Service-level routing
    * Escalation policy support
    * Auto-resolution on recovery
  </Card>

  <Card title="Email" icon="envelope" color="#EA4335">
    Send alert notifications via email to specific recipients.

    **Key Features:**

    * Multiple recipients
    * Detailed alert information
    * HTML-formatted messages
    * Recovery notifications support
    * Direct links to details
  </Card>

  <Card title="Webhook" icon="webhook" color="#FF6B6B">
    Send alerts to custom endpoints for integration with any system.

    **Key Features:**

    * Custom HTTP POST requests
    * JSON payloads
    * Trigger & recovery support
    * Configurable endpoints
  </Card>

  <Card title="AI Investigation" icon="robot" color="#8B5CF6">
    Automatically trigger AI investigations for firing alerts.

    **Key Features:**

    * Automated root cause analysis
    * Alert correlation
    * Remediation suggestions
    * Workflow integration
  </Card>
</CardGroup>

[Learn more about AI alert investigations →](/docs/ai-sre/alert-investigations)

## Configuring Alert Destinations

<Steps>
  <Step title="Navigate to Alert Configuration">
    Open your alerts page in the Metoro, then select the alert you want to configure destinations for, or create a new alert.
  </Step>

  <Step title="Add Destination">
    In the "Select Destination" section, click the **Add Destination** button
  </Step>

  <Step title="Choose Destination Type">
    Select your preferred destination type from the dropdown menu
  </Step>

  <Step title="Configure Settings">
    Enter the destination-specific settings (channel, email, webhook, etc.)
  </Step>

  <Step title="Save Configuration">
    Save your alert configuration to activate the destination
  </Step>
</Steps>

<Frame caption="Alert destination selector interface">
  <img src="https://mintcdn.com/metoro/ZLKLeijcjMYiIwum/images/alert-destination-selector.png?fit=max&auto=format&n=ZLKLeijcjMYiIwum&q=85&s=7b86b42fddf8189e08f2276acba01c5e" alt="Alert Destination Selector" width="1680" height="290" data-path="images/alert-destination-selector.png" />
</Frame>

### Multiple Destinations

<Tip>
  You can configure multiple destinations for a single alert to ensure notifications reach all necessary channels and teams.
</Tip>

<Tabs>
  <Tab title="Adding Multiple">
    * Click **Add Destination** for each additional destination
    * Each destination operates independently
    * Mix different destination types as needed
  </Tab>

  <Tab title="Managing">
    * Remove destinations using the X button
    * Edit existing destinations inline
  </Tab>

  <Tab title="Best Practices">
    * Use Slack for team visibility
    * Add PagerDuty or another incident response tooling's webhook for critical alerts
    * Configure email as backup
  </Tab>
</Tabs>

<Frame caption="Multiple destinations configured for comprehensive alerting">
  <img src="https://mintcdn.com/metoro/WUkDA8ZzUF2HFKec/images/multiple_destinations.png?fit=max&auto=format&n=WUkDA8ZzUF2HFKec&q=85&s=b9cbc60a573a7c4c0690cc9b47e66910" alt="Configure multiple alert destinations" width="1680" height="470" data-path="images/multiple_destinations.png" />
</Frame>

## Recovery Notifications

<Note>
  Recovery notifications are available for webhook, Slack, and email destinations, allowing you to know when alerts return to normal state.
</Note>

### What are Recovery Notifications?

Recovery notifications inform you when an alert condition is no longer met and the alert has returned to a normal state.

<AccordionGroup>
  <Accordion title="Benefits of Recovery Notifications" icon="check">
    * **Automated incident resolution** in external systems
    * **Clear communication** about alert lifecycle
    * **Reduced manual intervention** for ops teams
    * **Better tracking** of alert duration and impact
    * **Improved MTTR** (Mean Time To Recovery) metrics
  </Accordion>

  <Accordion title="How It Works" icon="gear">
    1. Alert triggers and sends notification to "firing" webhook
    2. Metoro continuously monitors the alert condition
    3. When condition is no longer met, alert recovers
    4. Recovery notification sent to "resolved" webhook
    5. External systems can auto-close incidents
  </Accordion>
</AccordionGroup>

<Tip>
  Include `$alert_uuid` and `$alert_state` variable in your webhook payload to differentiate between firing and resolved states for the same alert.
</Tip>

<Tip>
  Include `$fire_alert_uuid` variable in your webhook payload to differentiate between different firing instances of the same alert.
  This is useful is you have a `group by` in your alert configuration because alerts with a `group by` can have multiple firing instances.
  For example, if you have an alert that is grouped by with `service.name`, each service will have its own firing instance with a unique `$fire_alert_uuid`.
</Tip>

### Configuring Recovery Notifications

<Tabs>
  <Tab title="Slack">
    <Steps>
      <Step title="Add a Slack Destination">
        Select **Slack** from the destination type dropdown and choose your channel
      </Step>

      <Step title="Enable Recovery Notifications">
        Check the **"Notify when alarm recovers"** checkbox that appears next to the Slack destination
      </Step>
    </Steps>

    <Info>
      Slack recovery notifications are sent to the same channel as firing notifications, with clear visual indicators (✅ for resolved vs 🔥 for firing).
    </Info>
  </Tab>

  <Tab title="Email">
    <Steps>
      <Step title="Add an Email Destination">
        Select **Email** from the destination type dropdown and enter recipient email addresses
      </Step>

      <Step title="Enable Recovery Notifications">
        Check the **"Notify when alarm recovers"** checkbox that appears next to the email destination
      </Step>
    </Steps>

    <Info>
      Email recovery notifications are sent to the same recipients as firing notifications, with a blue theme and "ALERT RESOLVED" badge to clearly indicate the alert has returned to normal.
    </Info>
  </Tab>

  <Tab title="Webhook">
    <Steps>
      <Step title="Add a Webhook Destination">
        Select **Webhook** from the destination type dropdown and choose your primary webhook for alert triggers
      </Step>

      <Step title="Enable Recovery Notifications">
        Check the **"Notify when alarm recovers"** checkbox that appears next to the webhook destination.
        <Warning> If you have not selected a firing destination, you cannot select Notify when alarm recovers option.</Warning>
      </Step>

      <Step title="Configure Recovery Webhook">
        The interface expands to show two webhook selectors.
        You can use the same webhook for both or configure different endpoints.
      </Step>
    </Steps>
  </Tab>
</Tabs>

<Frame caption="Webhook recovery notification configuration showing both firing and resolved webhook options">
  <img src="https://mintcdn.com/metoro/WUkDA8ZzUF2HFKec/images/multiple_destinations.png?fit=max&auto=format&n=WUkDA8ZzUF2HFKec&q=85&s=b9cbc60a573a7c4c0690cc9b47e66910" alt="Recovery Notification Configuration" width="1680" height="470" data-path="images/multiple_destinations.png" />
</Frame>

### Recovery Webhook Payload

<CodeGroup>
  ```json Resolved Payload theme={null}
  {
    "alert_state": "resolved",
    "resolved_at": "2024-01-15T10:30:00Z",
    "alert_uuid": "abc-123",
    "alert_name": "High CPU Usage",
    "environment": "production",
    "service": "api-server"
  }
  ```

  ```json Firing Payload theme={null}
  {
    "alert_state": "firing",
    "alert_uuid": "abc-123",
    "alert_name": "High CPU Usage",
    "environment": "production",
    "service": "api-server",
  }
  ```
</CodeGroup>

<Warning>
  The `alert_state` field is critical for distinguishing between trigger and recovery notifications. Always validate this field in your webhook handler.
</Warning>

Please see the webhook documentation for more details on how to configure these payloads: [Webhook Documentation](/docs/integrations/webhooks).

## Troubleshooting

You can check webhook delivery status and errors in Metoro Logs view. This provides visibility into successful deliveries and any issues encountered.
Go to the [Logs page](https://us-east.metoro.io/logs) and filter by `environment=metoro-internal` and `service.name=metoro-webhooks` to see webhook notifications that are successfully sent or failed with errors.

<AccordionGroup>
  <Accordion title="Configured Webhook Payload is not sent" icon="circle-xmark">
    <Warning>
      Ensure that the webhook payload is a valid JSON object. You might be missing a closing brace or have an extra/missing comma.
    </Warning>
  </Accordion>

  <Accordion title="Authentication Errors" icon="lock">
    Common authentication issues and solutions:

    * **401 Unauthorized**: Check API keys/tokens. Make sure that your webhook includes **Authorization** header with **Bearer \<valid-token>** value.
    * **403 Forbidden**: Verify IP whitelisting
    * **Expired Token**: Refresh authentication tokens from your webhook provider
  </Accordion>
</AccordionGroup>

## Quick Reference

<CardGroup cols={2}>
  <Card title="API Documentation" icon="code" href="/docs/api-reference/alerts/createupdate-alert">
    Programmatically configure alert destinations using our REST API
  </Card>

  <Card title="Creating Alerts" icon="plus" href="./create_alert">
    Step-by-step guide to creating your first alert
  </Card>

  <Card title="Alert Examples" icon="lightbulb" href="./example_alerts">
    Common alert patterns and configurations
  </Card>

  <Card title="Integrations Setup" icon="plug" href="/docs/integrations/overview">
    Configure Slack, PagerDuty, webhooks, and more
  </Card>
</CardGroup>

***

<Note>
  **Need Help?**

  Join our [Slack community](https://join.slack.com/t/metorocommunity/shared_invite/zt-2makpjl5j-F0WcpGnPcdc8anbNGcewqw) for support, or contact [support@metoro.io](mailto:support@metoro.io) for assistance with alert destinations.
</Note>