> ## Documentation Index
> Fetch the complete documentation index at: https://metoro.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Traces OpenTelemetry Integration

This guide explains how to send distributed traces to Metoro using OpenTelemetry. Metoro supports the OpenTelemetry Protocol (OTLP) for trace ingestion, allowing you to send traces from any application or service that can export OpenTelemetry traces.

## Prerequisites

* A Metoro account
* An application configured with OpenTelemetry

## Pricing

Custom traces are billed at \$0.30 per GB.

## High Level Overview

The Metoro exporter running in each cluster accepts traces over OTLP/HTTP using protobuf or JSON payloads. The standard environment-variable configuration below uses the `http/protobuf` protocol.

## Endpoint Configuration

Use the following OTLP base URL when your SDK or collector automatically appends the signal-specific path:

```
http://metoro-exporter.metoro.svc.cluster.local/api/v1/custom/otel
```

Alternatively, use the full traces endpoint when configuring a signal-specific URL:

```
http://metoro-exporter.metoro.svc.cluster.local/api/v1/custom/otel/v1/traces
```

These endpoints use OTLP over HTTP and are available within the Kubernetes cluster where the Metoro exporter is installed. The Metoro exporter does not expose an OTLP/gRPC endpoint.

## Authentication

No additional authentication is required when sending traces from within the cluster to the Metoro exporter.

## OpenTelemetry Collector Configuration

If you're using the OpenTelemetry Collector to forward traces to Metoro, here's an example configuration:

```yaml theme={null}
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000

exporters:
  otlphttp/metoro:
    endpoint: http://metoro-exporter.metoro.svc.cluster.local/api/v1/custom/otel

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/metoro]
```

This configuration:

* Receives traces via OTLP over both HTTP (4318) and gRPC (4317)
* Batches traces for efficient transmission
* Forwards traces to the Metoro exporter via OTLP/HTTP

Ports `4317` and `4318` belong to the collector configured in this example. The collector forwards telemetry to the Metoro exporter through its in-cluster HTTP service.

## Language-Specific Examples

### Go

```go theme={null}
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer() (*sdktrace.TracerProvider, error) {
    ctx := context.Background()

    exporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint("metoro-exporter.metoro.svc.cluster.local"),
        otlptracehttp.WithURLPath("/api/v1/custom/otel/v1/traces"),
        otlptracehttp.WithInsecure(), // Since we're in-cluster
    )
    if err != nil {
        return nil, err
    }

    resource := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName("your-service-name"),
    )

    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource),
        sdktrace.WithSampler(sdktrace.AlwaysSample()),
    )
    otel.SetTracerProvider(tracerProvider)

    return tracerProvider, nil
}
```

### Python

```python theme={null}
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def init_tracer():
    exporter = OTLPSpanExporter(
        endpoint="http://metoro-exporter.metoro.svc.cluster.local/api/v1/custom/otel/v1/traces"
    )

    resource = Resource.create({
        "service.name": "your-service-name"
    })

    provider = TracerProvider(resource=resource)
    processor = BatchSpanProcessor(exporter)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)
    
    return provider
```

### Node.js

```javascript theme={null}
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');

const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'your-service-name',
});

const traceExporter = new OTLPTraceExporter({
  url: 'http://metoro-exporter.metoro.svc.cluster.local/api/v1/custom/otel/v1/traces',
  headers: {},
});

const provider = new NodeTracerProvider({
  resource: resource,
});

provider.addSpanProcessor(new BatchSpanProcessor(traceExporter));
provider.register();
```

### Via environment variables

Most OpenTelemetry SDKs support the generic OTLP environment variables. The SDK appends `/v1/traces` to the base URL:

```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT="http://metoro-exporter.metoro.svc.cluster.local/api/v1/custom/otel"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
```

## Inspecting otel traces

You can use otel traces just like any other trace type. Just search for span attributes. If you want to see all non ebpf traces you can copy the following filter into the trace search, or click the following link: [filters applied](https://us-east.metoro.io/traces?startEnd=\&environment=\&service=\&excludeFilter=%7B%22metoro.is_server_span%22%3A%5B%22regex%3A.%2B%22%5D%7D).

```json theme={null}
{"key":"metoro.is_server_span","values":["regex:.+"],"isExclude":true}
```

## Attributes and Context

When sending traces via OpenTelemetry, you can include additional attributes that will be searchable in Metoro:

* Use resource attributes to define static information about the service
* Use span attributes to include dynamic information with each span
* Link spans with logs using trace context propagation
* Add custom attributes to spans for business-specific data

## Troubleshooting

If you encounter issues with OpenTelemetry trace ingestion:

1. Verify your endpoint URL is correct
2. Check your network connectivity to the Metoro OTLP endpoint
3. Enable debug logging in your OpenTelemetry SDK
4. Verify your traces appear in the Metoro traces view
5. Contact support if issues persist

## Additional Resources

* [OpenTelemetry Documentation](https://opentelemetry.io/docs/)
* [OpenTelemetry Specification](https://github.com/open-telemetry/opentelemetry-specification)