> ## Documentation Index
> Fetch the complete documentation index at: https://metoro.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Traces Cheat Sheet

> A beginner-friendly guide to reading traces in Metoro and answering simple traffic questions

<Note>
  Start here if traces feel confusing. This page explains the minimum you need to answer questions like "who is calling this service?", "which endpoints are busiest?", and "what is failing?".
</Note>

## The 30-second version

* A trace is one request or one network call.
* `client.service.name` is the service that made the call.
* `server.service.name` is the service that received the call.
* `http.path` is the endpoint or route.
* `count(traces)` means "how many calls happened?"
* `by (...)` means "split that total into buckets."

## First: switch to MetoroQL mode

Most of the examples below are written in MetoroQL first, because it is the clearest way to express trace questions.

1. In **Trace Search**, click the `<>` icon at the end of the search section (at the right end of the search bar). The tooltip says `Switch to MetoroQL mode`.
2. In **Metric Explorer**, open the mode selector at the top-right of the query row and switch from `Standard mode` to `MetoroQL mode`.
3. If you want to come back later, use the same controls to switch back to `Standard mode`. Switching between the Standard and MetoroQL modes will convert the query automatically.

<Note>
  The "Standard Mode" examples later on refer to **Metric Explorer** with `trace` data selected. Trace Search standard mode is for browsing individual traces, not for grouped time-series questions.
</Note>

## One request, pictured simply

Keep this picture in your head for almost every traces question:

```mermaid theme={null}
%%{init: {
  'theme': 'dark',
  'themeVariables': {
    'fontFamily': 'Inter',
    'primaryColor': '#151F32',
    'primaryTextColor': '#EBF1F7',
    'primaryBorderColor': '#3793FF',
    'lineColor': '#334670',
    'secondaryColor': '#151F32',
    'tertiaryColor': '#151F32',
    'mainBkg': '#151F32',
    'nodeBorder': '#3793FF',
    'clusterBkg': '#182338',
    'titleColor': '#EBF1F7',
    'edgeLabelBackground': '#151F32',
    'clusterBorder': '#334670'
  }
}}%%
flowchart LR
    subgraph CLIENT["Source side (client)"]
        A["Service A<br/>caller<br/><br/>client.service.name = service-a"]
    end

    CALL["HTTP request span<br/><br/>http.method = GET<br/>http.path = /api/dev<br/>http.status_code = 200<br/>http.status_code.bucket = 2XX"]

    subgraph SERVER["Destination side (server)"]
        B["Service B<br/>receiver<br/><br/>server.service.name = service-b<br/>server-side view: metoro.is_server_span = true"]
    end

    A --> CALL --> B

    classDef service fill:#182338,stroke:#3793FF,color:#EBF1F7,stroke-width:2px;
    classDef request fill:#18312C,stroke:#2DD881,color:#EBF1F7,stroke-width:2px;
    class A,B service;
    class CALL request;
```

## Incoming vs outgoing, in plain English

If `service-a` calls `service-b` on `/api/dev`:

* From `service-a`'s point of view, that is an **outgoing** call.
* From `service-b`'s point of view, that is an **incoming** call.
* In Metoro, the caller is the `client.service.name`.
* In Metoro, the receiver is the `server.service.name`.

In tracing language:

* `client` does **not** necessarily mean a browser or end user. It means "the thing that started the request."
* `server` does **not** necessarily mean a VM or physical server. It means "the thing that received the request."

That leads to one simple rule:

* Want **outgoing calls from a service**: filter on `client.service.name`
* Want **incoming calls to a service**: filter on `server.service.name`

## Which field do I filter?

```mermaid theme={null}
%%{init: {
  'theme': 'dark',
  'themeVariables': {
    'fontFamily': 'Inter',
    'primaryColor': '#151F32',
    'primaryTextColor': '#EBF1F7',
    'primaryBorderColor': '#3793FF',
    'lineColor': '#334670',
    'secondaryColor': '#151F32',
    'tertiaryColor': '#151F32',
    'mainBkg': '#151F32',
    'nodeBorder': '#3793FF',
    'clusterBkg': '#182338',
    'titleColor': '#EBF1F7',
    'edgeLabelBackground': '#151F32',
    'clusterBorder': '#334670'
  }
}}%%
flowchart LR
    Q1["I care about calls leaving Service A"] --> F1["Filter on<br/>client.service.name = service-a"]
    Q2["I care about calls arriving at Service B"] --> F2["Filter on<br/>server.service.name = service-b"]

    classDef question fill:#182338,stroke:#3793FF,color:#EBF1F7,stroke-width:2px;
    classDef filter fill:#18312C,stroke:#2DD881,color:#EBF1F7,stroke-width:2px;
    class Q1,Q2 question;
    class F1,F2 filter;
```

## What the main trace fields mean

| Field                     | Plain English                                            |
| ------------------------- | -------------------------------------------------------- |
| `client.service.name`     | Who made the call                                        |
| `server.service.name`     | Who received the call                                    |
| `http.method`             | GET, POST, PUT, DELETE, etc.                             |
| `http.path`               | Which endpoint was hit                                   |
| `http.status_code`        | The exact response code                                  |
| `http.status_code.bucket` | The response code family, like `2XX` or `5XX`            |
| `metoro.is_server_span`   | Whether Metoro is showing the server side of the request |

## Other useful trace attributes

| Field                           | Plain English                                                |
| ------------------------------- | ------------------------------------------------------------ |
| `client.namespace`              | Kubernetes namespace that made the call                      |
| `server.namespace`              | Kubernetes namespace that received the call                  |
| `client.container.id`           | Exact caller pod/container                                   |
| `server.container.id`           | Exact receiver pod/container                                 |
| `server.net.host.name`          | Destination host or IP, especially useful for external calls |
| `server.external`               | Whether the destination is outside your Kubernetes cluster   |
| `environment`                   | Metoro environment                                           |
| `traceId`                       | Unique ID for one trace                                      |
| `client.host.availability_zone` | Availability zone the request came from                      |
| `server.host.availability_zone` | Availability zone that handled the request                   |

## Which screen to use

| If you want to...                                   | Use...                                                                      |
| --------------------------------------------------- | --------------------------------------------------------------------------- |
| See who talks to who                                | [Service Map](/docs/traces/service-map)                                          |
| Inspect individual requests                         | The [Traces page](/docs/traces/overview)                                         |
| Count requests, split by endpoint, or chart traffic | Metric Explorer in Standard Mode, or a MetoroQL query using `count(traces)` |

## Group by, without the jargon

`group by` just means:

> Do not give me one big total. Split the total into smaller buckets.

### No group by: one total

```mermaid theme={null}
%%{init: {
  'theme': 'dark',
  'themeVariables': {
    'fontFamily': 'Inter',
    'primaryColor': '#151F32',
    'primaryTextColor': '#EBF1F7',
    'primaryBorderColor': '#3793FF',
    'lineColor': '#334670',
    'secondaryColor': '#151F32',
    'tertiaryColor': '#151F32',
    'mainBkg': '#151F32',
    'nodeBorder': '#3793FF',
    'clusterBkg': '#182338',
    'titleColor': '#EBF1F7',
    'edgeLabelBackground': '#151F32',
    'clusterBorder': '#334670'
  }
}}%%
flowchart LR
    FILTER1["Filter traces<br/>server.service.name = service-b"] --> GROUP1["Group by<br/>leave empty"] --> TOTAL["Result<br/>100 total calls"]

    classDef step fill:#182338,stroke:#3793FF,color:#EBF1F7,stroke-width:2px;
    classDef result fill:#18312C,stroke:#2DD881,color:#EBF1F7,stroke-width:2px;
    class FILTER1,GROUP1 step;
    class TOTAL result;
```

```PromQL theme={null}
count(traces{server.service.name="service-b"})
```

### Group by `http.path`: split by endpoint

```mermaid theme={null}
%%{init: {
  'theme': 'dark',
  'themeVariables': {
    'fontFamily': 'Inter',
    'primaryColor': '#151F32',
    'primaryTextColor': '#EBF1F7',
    'primaryBorderColor': '#3793FF',
    'lineColor': '#334670',
    'secondaryColor': '#151F32',
    'tertiaryColor': '#151F32',
    'mainBkg': '#151F32',
    'nodeBorder': '#3793FF',
    'clusterBkg': '#182338',
    'titleColor': '#EBF1F7',
    'edgeLabelBackground': '#151F32',
    'clusterBorder': '#334670'
  }
}}%%
flowchart LR
    FILTER2["Filter traces<br/>server.service.name = service-b"] --> GROUP2["Group by<br/>http.path"] --> P1["/api/dev<br/>90 calls"]
    GROUP2 --> P2["/api/health<br/>10 calls"]

    classDef step fill:#182338,stroke:#3793FF,color:#EBF1F7,stroke-width:2px;
    classDef result fill:#18312C,stroke:#2DD881,color:#EBF1F7,stroke-width:2px;
    class FILTER2,GROUP2 step;
    class P1,P2 result;
```

```PromQL theme={null}
count(traces{server.service.name="service-b"}) by (http.path)
```

In the platform UI, putting `http.path` in the `Group by` control does the same thing as writing `by (http.path)` in MetoroQL.

Add more fields when you need narrower buckets. For example, `by (http.method, http.path)` splits `GET /api/dev` and `POST /api/dev` into separate rows.

Examples:

```PromQL theme={null}
# One total number of incoming calls
count(traces{server.service.name="<service>"})

# Split incoming calls by endpoint
count(traces{server.service.name="<service>"}) by (http.path)

# Split incoming calls by method and endpoint
count(traces{server.service.name="<service>"}) by (http.method, http.path)
```

If multiple services have the same endpoint name, include the service in the `by (...)` clause too.

## Copy-paste questions and answers

Each example below has:

* a MetoroQL version you can paste directly
* a Standard Mode version you can build in Metric Explorer without writing the query yourself

### Number of incoming calls to a service

```PromQL theme={null}
count(traces{server.service.name="<service>"})
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `server.service.name=<service>`
* `Group by`: leave empty

### Number of incoming calls to a service by endpoint

```PromQL theme={null}
count(traces{server.service.name="<service>"}) by (http.path)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `server.service.name=<service>`
* `Group by`: `http.path`

### Number of incoming calls to a service by method and endpoint

```PromQL theme={null}
count(traces{server.service.name="<service>"}) by (http.method, http.path)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `server.service.name=<service>`
* `Group by`: `http.method`, `http.path`

### Number of outgoing calls from a service

```PromQL theme={null}
count(traces{client.service.name="<service>"})
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `client.service.name=<service>`
* `Group by`: leave empty

### Number of outgoing calls from a service by destination service

```PromQL theme={null}
count(traces{client.service.name="<service>"}) by (server.service.name)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `client.service.name=<service>`
* `Group by`: `server.service.name`

### Number of outgoing calls from a service by endpoint

```PromQL theme={null}
count(traces{client.service.name="<service>"}) by (server.service.name, http.path)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `client.service.name=<service>`
* `Group by`: `server.service.name`, `http.path`

### Which services are calling my service?

```PromQL theme={null}
count(traces{server.service.name="<service>"}) by (client.service.name)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `server.service.name=<service>`
* `Group by`: `client.service.name`

### Which namespaces are calling my service?

```PromQL theme={null}
count(traces{server.service.name="<service>"}) by (client.namespace)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `server.service.name=<service>`
* `Group by`: `client.namespace`

### All endpoint call counts in a namespace

```PromQL theme={null}
count(traces{server.namespace="<namespace>"}) by (server.service.name, http.path)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `server.namespace=<namespace>`
* `Group by`: `server.service.name`, `http.path`

### Which endpoints on my service are failing?

```PromQL theme={null}
count(traces{server.service.name="<service>", http.status_code.bucket="5XX"}) by (http.path)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `server.service.name=<service>`, `http.status_code.bucket=5XX`
* `Group by`: `http.path`

### Which endpoints on my service are slow?

```PromQL theme={null}
trace_duration_quantile(0.95, traces{server.service.name="<service>"}) by (http.path)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `p95 latency`
* `Filters`: `server.service.name=<service>`
* `Group by`: `http.path`

### Which containers in my service are handling traffic?

```PromQL theme={null}
count(traces{server.service.name="<service>"}) by (server.container.id)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `server.service.name=<service>`
* `Group by`: `server.container.id`

### Which external APIs is my service calling?

```PromQL theme={null}
count(traces{client.service.name="<service>", server.namespace="External Service"}) by (server.service.name, http.path)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `client.service.name=<service>`, `server.namespace=External Service`
* `Group by`: `server.service.name`, `http.path`

### Which external hosts is my service calling?

```PromQL theme={null}
count(traces{client.service.name="<service>", server.namespace="External Service"}) by (server.net.host.name, http.path)
```

**Standard Mode**

* `Timeseries data`: `trace`
* `Stat`: `request count`
* `Filters`: `client.service.name=<service>`, `server.namespace=External Service`
* `Group by`: `server.net.host.name`, `http.path`

## A real mental model

If you are looking at **your API service**:

* `server.service.name="<your-api>"` means "requests coming into my API"
* `client.service.name="<your-api>"` means "requests my API made to something else"

If you only remember one thing, remember that.

## If you do not know the exact service name

Service names in Metoro are often full Kubernetes-style names such as `/k8s/default/checkout-service`.

If you are not sure of the exact value, use a regex match:

```PromQL theme={null}
count(traces{server.service.name=~".*checkout-service.*"}) by (http.path)
```

## When `http.path` is not enough

`http.path` is best for HTTP traffic. For other protocols, use a more useful field for that protocol:

* `span.name` for a generic span name
* `db.operation` for database calls
* `server.service.name` to see which downstream service was hit

## Beginner workflow

<Steps>
  <Step title="Start with the service you care about">
    Decide whether you care about traffic coming **into** the service or going **out of** the service.
  </Step>

  <Step title="Pick the right side">
    Use `server.service.name` for incoming traffic. Use `client.service.name` for outgoing traffic.
  </Step>

  <Step title="Add one split at a time">
    Start with `by (http.path)`. If that is too broad, use `by (http.method, http.path)` or `by (server.service.name, http.path)`.
  </Step>

  <Step title="Add failure or latency next">
    Add `http.status_code.bucket="5XX"` to focus on errors, or switch to `trace_duration_quantile(0.95, ...)` to look for slow endpoints.
  </Step>
</Steps>

## Related docs

<CardGroup cols={3}>
  <Card title="Traces Overview" icon="route" href="/docs/traces/overview">
    Understand how trace data is collected and shown in Metoro
  </Card>

  <Card title="Service Map" icon="diagram-project" href="/docs/traces/service-map">
    See who talks to who before drilling into specific requests
  </Card>

  <Card title="MetoroQL" icon="rectangle-terminal" href="/docs/metrics/metoroql">
    Learn the query syntax behind the examples on this page
  </Card>
</CardGroup>