Structured Logs

Metoro provides advanced parsing capabilities for structured logs, automatically extracting and indexing fields from various log formats. This makes your structured logs fully searchable and helps you get more value from your logging data.

​

JSON Structured Logs

Metoro automatically detects and parses JSON-formatted logs. When a log entry is in JSON format, Metoro will:

1. Extract all fields from the JSON object
2. Flatten nested JSON structures using dot notation
3. Index all fields for searching
4. Handle the message field specially

For example, if your log entry is:

{
  "service": "payment-processor",
  "region": "us-west",
  "error": {
    "code": 500,
    "details": "Database connection failed"
  },
  "msg": "Transaction processing failed"
}

Metoro will:

1. Extract and index these fields:
   • service: "payment-processor"
   • region: "us-west"
   • error.code: "500"
   • error.details: "Database connection failed"
2. Use the msg field as the main log message
3. Make all fields searchable using attribute filters

You can then search for these logs using attribute filters like:

• error.code = "500"
• service = "payment-processor"
• error.details = regex: .*connection.*

​

Message Field Handling

For JSON-formatted logs, Metoro looks for a dedicated message field in this order:

1. msg field
2. message field
3. If neither exists, the entire JSON object is preserved as the log body

For example, this JSON log:

{
  "timestamp": "2024-03-15T10:30:00Z",
  "level": "error",
  "service": "order-service",
  "msg": "Failed to process order",
  "order_id": "12345",
  "error_code": 500
}

Will be displayed as:

• Log Message: “Failed to process order”
• Log Attributes:
  • timestamp: "2024-03-15T10:30:00Z"
  • level: "error"
  • service: "order-service"
  • order_id: "12345"
  • error_code: "500"

This makes your logs more readable while keeping all fields searchable.

​

LogZero Format

Metoro also supports the LogZero format, which follows this pattern:

[LEVEL DATE TIME module:line] message

For example:

[I 250313 16:24:23 my_handler:160] Request processed successfully

When parsing LogZero format, Metoro extracts:

• level: Log level (I=info, D=debug, W=warning, E=error, C=critical)
• module: The module name
• line: The line number
• Remaining text:
  • Becomes the log message if the message is not JSON-formatted
  • Is parsed as JSON if the message is JSON-formatted

These fields are then indexed and made searchable like any other log attribute.

​

Best Practices

1. Use Consistent Formats: Stick to a consistent log format across your services
2. Include Essential Fields: Always include:
   • Timestamp
   • Service name
   • Log level/severity
   • A clear message field
3. Structured Data: Use JSON formatting for logs when possible
4. Nested Data: Feel free to use nested JSON objects - Metoro will flatten them automatically
5. Field Naming: Use clear, consistent field names across your services

​

Searching Structured Logs

You can search through structured log fields using:

1. Exact matches: field = "value"
2. Regex matches: field = regex: pattern
3. Multiple values: field = ["value1", "value2"]
4. Nested fields: parent.child = "value"

For example:

error.code = "500"
service.name = regex: .*api.*
environment = ["prod", "staging"]