Building an AI SRE with Human Approval Built In

Suppose you wanted to build your own AI SRE. You'd want it to investigate an alert, work out what's probably broken, and do something useful about it. Ideally, the on-call engineer would get a proposed fix with enough evidence to make a decision, rather than another place to start searching.

The model is one part of that. It also needs a way to find out what's happening in production, and a clear boundary around what it's allowed to change. An agent with poor context can confidently fix the wrong thing. An agent that can only make suggestions still leaves someone else doing all the work.

For this example, Metoro supplies production context through its public interfaces, withHuman handles approval of tool calls, and you build the agent and remediation tools. The architecture, investigation workflow, and executor described below are an illustrative design for your own AI SRE. They do not describe how Metoro's own AI SRE is implemented.

Start with one incident

Start with a narrow job: investigate a rise in errors after a Kubernetes deployment and propose a rollback when the evidence supports it.

That gives us something concrete to design around. Did the errors actually start with the release? Are they confined to the changed workload? Is the failing service responsible, or is it reporting a failure from a dependency? Would a rollback still be compatible with the current database schema?

For this example, imagine payments-api starts returning more 5xx responses after a deployment. The agent needs to gather evidence, test that explanation, and produce a specific action. It should also be able to conclude that it hasn't found enough evidence to act.

Your example architecture could look like this:

flowchart TD
  A["Alert: payments-api errors increased"] --> R["Your agent runtime"]
  R --> C["Metoro MCP: production context"]
  C --> R
  R --> P["Proposed remediation tool call"]
  P --> W["withHuman: apply approval rules"]
  W -->|Risky action| H["Service owner reviews evidence and change"]
  H -->|Approved| E["Your executor: validate and apply"]
  H -->|Denied or expired| S["Stop and report"]
  W -->|Denied by policy| S
  W -->|Automatically approved| E
  E --> V["Query Metoro again: did the service recover?"]

Give it a way to investigate

Metoro collects logs, metrics, traces, and Kubernetes context. Its MCP server exposes tools an external agent can use to query that data. Connect it to your runtime with a Metoro API key and the API URL for your workspace. The setup guide covers building the MCP server and supplying METORO_AUTH_TOKEN and METORO_API_URL.

Start with a small set of read tools. Metoro's MCP interface also includes operations that create or update records, so importing its entire tool catalogue isn't the same as giving an agent read-only access. Choose the tools explicitly and give the API key only the permissions they need.

For the payments incident, your context adapter needs to answer a few concrete questions:

Have the adapter use the tool schemas exposed by your MCP connection and discover available filter attributes before querying. Keep the mapping between your incident model and those tool arguments in your own integration code.

You might represent a query scope in your application like this. This is an example schema for your adapter, not a Metoro API request:

{
  "cluster": "production-eu",
  "namespace": "payments",
  "workload": "payments-api",
  "incident_window": {
    "start": "2026-10-26T10:10:00Z",
    "end": "2026-10-26T10:20:00Z"
  },
  "baseline_window": {
    "start": "2026-10-26T10:00:00Z",
    "end": "2026-10-26T10:10:00Z"
  }
}

These are illustrative values. The useful detail is the fixed time window. Compare it with a baseline before the deployment, and use the same incident window for related traces and logs. Otherwise, a slow investigation can end up comparing different slices of “the last ten minutes.”

Keep the queries bounded, preserve source references, and record what was missing. A small sample of error traces can help explain a failure, but it cannot establish the error rate. That needs an aggregate query with a request count as the denominator.

Make the agent show its working

The agent's investigation state should distinguish observations from explanations. “Errors increased after the deployment” is an observation. “The deployment caused the errors” is a hypothesis that needs more evidence.

In your agent's own state store, keep the affected workload, time windows, query results, evidence references, hypotheses, and unresolved questions for each incident. That lets the agent revisit a conclusion without losing why it reached it, and gives the human reviewer something more useful than a confident paragraph.

The runtime also needs a query budget, a deadline, and a stopping rule. If it cannot distinguish a bad release from a dependency outage within those limits, it should hand the investigation to an engineer. Endless tool calls aren't progress.

Logs and other retrieved content should stay in the evidence part of the conversation. A string inside an application log must never become an instruction to the agent or a reason to change its permissions.

Give it small, specific action tools

Once the investigation supports a fix, the agent needs a way to propose it. You could implement a tool called rollback_deployment, backed by an executor service you operate. This example tool is part of your agent, and its executor owns the Kubernetes write credential.

Before requesting approval, the tool adapter prepares a fixed proposal: cluster, namespace, Deployment UID, expected resource version, exact target pod template, and the evidence supporting the change. It can resolve the template from retained rollout history and show its diff to the reviewer. The implementation should preserve the full template, including sidecars, probes, and resource settings.

An instruction like “roll back to the previous version” leaves too much undecided. Another release might happen during review. The action that eventually runs must be the one the person saw.

Keep live Kubernetes state separate from investigation context. Historical resource snapshots from Metoro help explain the incident. The executor must check the current object before changing it.

Put withHuman around the tool calls

This is where withHuman's approval layer fits. An adapter using the Agent Approval Protocol (AAP) intercepts the call before the tool executes and asks withHuman for a decision. Your rules can let routine reads through, refuse unsupported operations, and send risky changes to a person.

For this example, suppose the team permits the agent to roll back deployments only in the payments namespace of production-eu. Configure the approval pipeline around that explicit scope:

withHuman isn't inferring which cluster is appropriate. You configure conditions on the tool name and its arguments, requiring all three checks in the rollback row to match. A rollback targeting staging-eu or another namespace falls through to deny, as does one missing either target field. If the action comes through MCP, also match the expected MCP server identity to distinguish tools with the same name.

The executor must enforce the same scope and map production-eu to a fixed Kubernetes endpoint and credential. Otherwise, the request could say one thing while the tool acts somewhere else. Order matters because a deciding block stops evaluation. Add the final deny explicitly; withHuman's fallback when no block decides is human review.

You can attach approvals at the framework level. For example, withHuman documents an AAP middleware for LangChain agents. If you choose LangChain for your own agent, its create_agent function lets you wire those pieces together like this:

import os

from agentapprovalprotocol.langgraph import ApprovalMiddleware
from langchain.agents import create_agent


def build_sre_agent(model, metoro_read_tools, action_tools):
    return create_agent(
        model=model,
        tools=[*metoro_read_tools, *action_tools],
        middleware=[
            ApprovalMiddleware(
                "https://app.withhuman.ai/api/aap",
                os.environ["WITHHUMAN_TOKEN"],
            )
        ],
        system_prompt=(
            "Investigate the incident using bounded telemetry queries. "
            "Cite evidence and distinguish observations from hypotheses. "
            "Propose a remediation only when the evidence supports it. "
            "After an action, verify recovery. If denied, stop and report."
        ),
    )

This example uses public SDKs and expects your model, selected MCP tools adapted for LangChain, and action tools you've implemented. You'll need to build the incident service around it. You also need a withHuman agent instance and configured policies; the integration guide covers the credentials and SDK installation.

The middleware covers both tool groups. Policy decides which calls need a human, so the model isn't responsible for deciding whether to ask. The runtime must prevent alternate execution paths, including unprotected subagents or a shell with access to cluster credentials.

Remember that somebody may take a while to answer

A human decision can take longer than the model call that produced the proposal. Your runtime should persist the incident state, proposal, and approval request ID so its worker can resume without starting the investigation again. Serialize mutations per workload to avoid two incidents racing to fix the same service.

The AAP lifecycle includes checking approval expiry and consuming an approval before execution. Pending, denied, expired, or cancelled requests must not reach the mutation tool. Reusing an approval for different arguments is out of bounds too.

After approval, the executor still has work to do. A conditional Kubernetes patch can test the reviewed UID and resource version before applying the template. If either changed, stop and prepare a new proposal. A strict resource-version check will also catch status updates; that's a conservative tradeoff you can refine as the implementation matures.

Persist the execution attempt before sending it. A lost response doesn't tell you whether Kubernetes applied the change. Read back state and reconcile the attempt before retrying. Approval consumption and a Kubernetes mutation are separate transactions.

The executor should have narrow Kubernetes permissions, validate the permitted fields, and hold credentials the agent cannot retrieve. For GitOps-managed workloads, make the change through the repository and reconciler that own desired state.

Check that it helped

An API response isn't the end of the incident. After the action, query Metoro again: did error rates fall, did latency recover, and is the workload healthy? Use a defined observation window and account for request volume. Zero errors during zero traffic doesn't demonstrate recovery.

Keep execution success and incident resolution as separate outcomes. If the rollback applied but the service is still failing, return to investigation or escalate. Approval for that rollback doesn't authorize another mutation.

Before enabling writes, run your agent against recorded incidents and compare its evidence and proposed actions with what engineers actually did. Include cases where the right answer was to do nothing: a dependency outage, missing telemetry, a release already superseded, or a rollback incompatible with a migration. Then exercise denial, timeout, stale-state, and worker-restart paths in a test cluster.

That gives you a useful first version: one kind of incident, a small set of context tools, and one remediation that a person can review. Add more actions as you learn where it works and where it needs help.

We're building withHuman alongside Metoro because these two pieces belong in the same conversation: agents need enough context to propose a useful change, and teams need control over whether that change happens. Your own AI SRE can use both while keeping its reasoning, runbooks, and execution tools in your hands.