Metoro provides granular control over which logs are sent from your cluster for ingestion. You can configure inclusion and exclusion patterns to manage your log data effectively.

​
Overview

Log ingestion settings allow you to:

  • Include only specific logs that match certain patterns
  • Exclude sensitive or unnecessary logs from being sent out of your cluster
  • Apply filters based on services or namespaces
  • Control log ingestion across different environments

​
Configuring Log Filters

You can configure log filters from the settings page under the Data Ingestion Settings -> Log Filters tab.

Each log filter consists of:

  1. Include/Exclude Pattern: Choose whether to include or exclude logs that match your filter
  2. Environment Selection: Apply the filter to specific environments or all environments
  3. Filter Type: Choose between filtering by specific services or namespaces
  4. Pattern Matching: Define regex patterns in re2 to match log content

​
Include Filters

Include filters allow you to specify which logs should be sent to Metoro. Only logs matching these patterns will be sent out of your cluster. This is useful when you want to:

  • Only collect logs from specific services
  • Only ingest logs containing certain patterns
  • Limit log ingestion to specific namespaces

​
Exclude Filters

Exclude filters prevent matching logs from being sent to Metoro. This is useful for:

  • Protecting sensitive data
  • Reducing noise from verbose logging
  • Managing ingestion costs
  • Filtering out unnecessary debug logs

​
Best Practices

  1. Start with Broad Patterns: Begin with broader patterns and refine them based on your needs
  2. Test Your Patterns: Use the regex pattern carefully to ensure you’re not accidentally excluding important logs
  3. Regular Review: Periodically review your filters to ensure they still align with your needs
  4. Adjust log alerts: If you have log alerts set up, make sure they are up to date with your log filters

​
Examples

Here are some common use cases for log filters:

  1. Exclude Sensitive Data
Type: Exclude
Environment: All Environments
Filter Type: Service
Services: /k8s/authorization/auth-service
Pattern: password=.*

This filter will exclude logs containing the pattern password=.* from the auth-service service in authorization namespace in all environments.

  1. Include Only Error Logs
Type: Include
Environment: All Environments
Pattern: .*ERROR.*|.*FATAL.*
Filter Type: Namespace
Namespaces: namespaceX

This filter will include logs containing the patterns ERROR or FATAL from all services in the namespaceX namespace in all environments.

  1. Exclude Debug Logs
Type: Exclude
Environment: dev
Pattern: .*DEBUG.*
Filter Type: Service
Services: All Services

This filter will exclude logs containing the pattern DEBUG from all services in the dev environment.

Was this page helpful?