​
Why is the trace destination showing as “Unknown External Service”?

When Metoro monitors calls from services inside your cluster to external services, it uses a combination of client-side tracing and DNS resolution to identify the destination services. Here’s how it works:

​
The Process

  1. Client-Side Tracing: Metoro monitors all calls made from inside your cluster to external services. For example, when Service A (inside your cluster) calls Service B (outside your cluster), through client-side traces, Metoro can see the IP address of the destination.

  2. Protocol-Specific Resolution:

    HTTP/HTTPS Protocols: For HTTP-based protocols, the destination hostname is present in the request headers. Metoro directly uses this hostname without needing additional DNS resolution. This provides accurate service identification for HTTP/HTTPS calls.

    Other Protocols (Redis, MongoDB, etc.): For protocols where hostname isn’t part of the protocol data, Metoro captures DNS resolution results when the connection is first established. This information is saved to a key-value store (Redis node running in your cluster). The stored mapping helps identify services in subsequent traces.

  3. IP Resolution Process: When exporters encounter a trace with an unknown IP address (external cluster IP), they attempt to resolve it by looking up the IP in the key-value store. If the IP address cannot be found in the key-value store, the service gets labeled as “External Unknown Service”.

The DNS resolution results are stored with different retention periods based on your tier:

  • Hobby tier: DNS results are stored for 1 hour
  • Scale tier: DNS results are stored for 1 week

This becomes important for long-lived connections, especially with protocols like Redis. For example, if you’re using Redis in the Hobby tier and your connection stays alive for more than an hour, subsequent traces might show “Unknown External Service”. This happens because the original DNS resolution has expired, but the connection is still active and no new DNS resolution has occurred.

If a container makes a request to an IP address without making a DNS query that resolves to that IP, it will be shown as “Unknown External Service”. Here’s an example:

import requests

# Example 1: HTTP request directly to IP
# This will show as "Unknown External Service" because no DNS query is made
response = requests.get("http://1.2.3.4:8080/api/data")

# Example 2: Using hostname
# This will show the proper service name because DNS resolution occurs
response = requests.get("http://api.example.com/api/data")

​
What are the different types of traces in Metoro?

Metoro captures two main types of traces to provide comprehensive visibility into your service communications.

​
Client-Side Traces

Client-side traces are generated by inspecting the outgoing system calls from services within your cluster. When Service A makes requests, Metoro monitors these calls at the system level to understand the destination IP address, the protocol being used, the timing of the request, and other relevant metadata.

​
Server-Side Traces

Server-side traces are generated when your services receive incoming requests. These traces provide information about the incoming request details, how your service processed the request, the response sent back, and performance metrics for the request handling. Server-side traces are particularly useful for understanding how your services handle incoming traffic.

By default, all traces you see are client-side traces when the client IP is within your cluster. However, for incoming calls from external sources (where the client IP is outside your cluster), Metoro automatically switches to using server-side traces.

​
How do I know if a trace is a client-side or server-side trace?

You can identify the type of trace by looking at the trace attributes. If the attribute metoro.is_server_span is set to true, then it’s a server-side trace. If this attribute is not present or set to false, then it’s a client-side trace.

​
How do I view logs for a specific trace or span?

Metoro now provides integrated log viewing directly within the trace view, making it easy to correlate logs with traces:

​
Viewing Trace-Level Logs

To see all logs associated with an entire trace:

  1. Open the trace you’re interested in
  2. Click the “View Logs” button next to the “Share Trace” button in the trace header
  3. The logs view will open filtered by the trace ID, showing all logs from the entire request flow

​
Viewing Span-Level Logs

To see logs for a specific span within a trace:

  1. Click on any span in the trace view
  2. In the span details panel that opens, click on the “Logs” tab
  3. The logs view will show only logs associated with that specific span ID

​
Linked Traces

For eBPF traces that are linked to OpenTelemetry traces, Metoro automatically uses the correct trace ID when filtering logs, ensuring you see all relevant log entries regardless of the tracing mechanism used.

​
Can I search for logs directly from the trace view?

Yes! The integrated logs view within traces supports all the same search and filtering capabilities as the main logs view, including:

  • Regex search patterns
  • Attribute-based filtering
  • Time range selection (automatically set based on the trace duration)
  • Full-text search within log messages

This integration eliminates the need to manually copy trace IDs and switch between different views when debugging issues.