​

Sending Logs with Vector

This guide demonstrates how to send logs to Metoro using Vector and the OpenTelemetry Collector. This setup provides a robust and scalable way to collect and forward logs to Metoro.

For a complete working example, see our Vector Logs Example repository.

​

Architecture

The setup consists of two main components:

1. Vector: Acts as the log collector and forwarder. In this example, it generates sample syslog messages and forwards them to the OpenTelemetry Collector.
2. OpenTelemetry Collector: Receives logs from Vector, processes them, and forwards them to Metoro in the correct OpenTelemetry format.

​

Vector Configuration

Vector needs to be configured to forward logs to the OpenTelemetry Collector. Here’s the sink configuration:

transforms:
  add_service_name:
    type: remap
    inputs: ["your_source_name"]
    source: |
      # Add a service name to help organize logs in Metoro
      .service.name = "your-service-name"

sinks:
  syslog_sink:
    type: socket
    inputs: ["add_service_name"]  # Use the transform output
    address: "otel-collector:1514"  # Address of your OpenTelemetry Collector
    mode: "tcp"
    encoding:
      codec: "text"

​

Vector Source Options

Vector supports many different sources for collecting logs:

• kubernetes_logs: Collect logs from Kubernetes containers
• file: Read logs from files
• syslog: Accept syslog messages
• journald: Read from systemd journal
• And many more

​

OpenTelemetry Collector Configuration

The OpenTelemetry Collector needs to be configured to receive logs from Vector and forward them to Metoro. Here’s the configuration:

receivers:
  # Configure the syslog receiver
  syslog:
    tcp:
      listen_address: 0.0.0.0:1514
    protocol: rfc5424

processors:
  # Batch logs for efficient processing
  batch:
    timeout: 1s
    send_batch_size: 4096

exporters:
  # Configure the Metoro exporter
    otlphttp:
      endpoint: "http://metoro-exporter.metoro.svc.cluster.local/api/v1/send/logs/otel"
      logs_endpoint: "http://metoro-exporter.metoro.svc.cluster.local/api/v1/send/logs/otel"
      insecure: true

service:
  # Set up the processing pipeline
  pipelines:
    logs:
      receivers: [syslog]
      processors: [batch]
      exporters: [otlphttp]

​

OpenTelemetry Collector Components

1. Receivers: Configure how the collector receives data

   • In this case, we use the syslog receiver on port 1514
   • Supports RFC5424 format syslog messages

2. Processors: Configure data processing

   • Batch processor groups logs for efficient sending
   • Adjustable timeout and batch size settings

3. Exporters: Configure where to send the data

   • Uses OTLP HTTP protocol to send to Metoro
   • Endpoint points to your Metoro instance

​

Customization

​

Log Sources

To collect logs from your applications, configure an appropriate Vector source for your use case. See the Vector documentation for available source types and their configurations.

​

Batch Settings

Adjust the batch processor settings in the OpenTelemetry Collector for your needs:

processors:
  batch:
    timeout: 1s  # Increase for higher latency tolerance
    send_batch_size: 4096  # Adjust based on log volume

​

Troubleshooting

Common issues and solutions:

1. Vector can’t connect to the OpenTelemetry Collector:

   • Verify the collector address is correct
   • Check that the collector is listening on the specified port
   • Ensure network connectivity between Vector and the collector

2. Logs not appearing in Metoro:

   • Check the OpenTelemetry Collector logs for errors
   • Verify the Metoro endpoint is correct
   • Check that logs are being received by the collector

​

Next Steps

• Configure Vector to collect your application logs
• Add filters and transforms in Vector to process logs
• Set up monitoring for Vector and the OpenTelemetry Collector
• Configure log retention and archival policies