This guide explains how to send distributed traces to Metoro using OpenTelemetry. Metoro supports the OpenTelemetry Protocol (OTLP) for trace ingestion, allowing you to send traces from any application or service that can export OpenTelemetry traces.

Prerequisites

  • A Metoro account
  • An application configured with OpenTelemetry

Pricing

Custom traces are billed at $0.30 per GB.

High Level Overview

The metoro exporter running in each cluster is a fully compliant OpenTelemetry collector. This means that you can send traces to Metoro using any OpenTelemetry compatible tracing library.

Endpoint Configuration

Configure your OpenTelemetry exporter to send traces to:

http://metoro-exporter.metoro.svc.cluster.local:8080/api/v1/traces/otel

This endpoint is available within your Kubernetes cluster where the Metoro exporter is installed.

Authentication

No additional authentication is required when sending traces from within the cluster to the Metoro exporter.

OpenTelemetry Collector Configuration

If you’re using the OpenTelemetry Collector to forward traces to Metoro, here’s an example configuration:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000

exporters:
  otlphttp:
    endpoint: http://metoro-exporter.metoro.svc.cluster.local:8080/api/v1/traces/otel
    tls:
      insecure: true # Since we're in-cluster

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

This configuration:

  • Receives traces via OTLP over both HTTP (4318) and gRPC (4317)
  • Batches traces for efficient transmission
  • Forwards traces to the Metoro exporter
  • Uses insecure communication since we’re within the cluster

Language-Specific Examples

Go

package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer() (*sdktrace.TracerProvider, error) {
    ctx := context.Background()

    exporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint("metoro-exporter.metoro.svc.cluster.local:8080"),
        otlptracehttp.WithURLPath("/api/v1/traces/otel"),
        otlptracehttp.WithInsecure(), // Since we're in-cluster
    )
    if err != nil {
        return nil, err
    }

    resource := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName("your-service-name"),
    )

    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource),
        sdktrace.WithSampler(sdktrace.AlwaysSample()),
    )
    otel.SetTracerProvider(tracerProvider)

    return tracerProvider, nil
}

Python

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def init_tracer():
    exporter = OTLPSpanExporter(
        endpoint="http://metoro-exporter.metoro.svc.cluster.local:8080/api/v1/traces/otel",
        insecure=True  # Since we're in-cluster
    )

    resource = Resource.create({
        "service.name": "your-service-name"
    })

    provider = TracerProvider(resource=resource)
    processor = BatchSpanProcessor(exporter)
    provider.add_span_processor(processor)
    trace.set_tracer_provider(provider)
    
    return provider

Node.js

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');

const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'your-service-name',
});

const traceExporter = new OTLPTraceExporter({
  url: 'http://metoro-exporter.metoro.svc.cluster.local:8080/api/v1/traces/otel',
  headers: {},
});

const provider = new NodeTracerProvider({
  resource: resource,
});

provider.addSpanProcessor(new BatchSpanProcessor(traceExporter));
provider.register();

Attributes and Context

When sending traces via OpenTelemetry, you can include additional attributes that will be indexed and searchable in Metoro:

  • Use resource attributes to define static information about the service
  • Use span attributes to include dynamic information with each span
  • Link spans with logs using trace context propagation
  • Add custom attributes to spans for business-specific data

Best Practices

  1. Sampling: Configure appropriate sampling rates based on your traffic volume
  2. Span Names: Use descriptive span names that identify the operation
  3. Error Handling: Set span status to error when exceptions occur
  4. Attributes: Add relevant attributes but avoid excessive data
  5. Context Propagation: Properly propagate trace context across service boundaries

Troubleshooting

If you encounter issues with OpenTelemetry trace ingestion:

  1. Verify your endpoint URL is correct
  2. Check your network connectivity to the Metoro OTLP endpoint
  3. Enable debug logging in your OpenTelemetry SDK
  4. Verify your traces appear in the Metoro traces view
  5. Contact support if issues persist

Additional Resources