MetoroQL

MetoroQL (mQL for short) is Metoro’s query language for observability data. It’s designed to be familiar to users of PromQL but with several important enhancements that make it more powerful for querying across different types of observability data.

Overview

MetoroQL has a PromQL-like syntax but provides unified access to different types of data:

Metrics (both standard and custom)
Logs
Traces
Kubernetes resources

This allows you to correlate and analyze data from different sources using a consistent query language.

Key Differences from PromQL

MetoroQL is generally a subset of promql with a few notable difference:

Counters return the delta of consecutive values by default
Queries can be over resource types other than metrics
Timeseries queries must have an aggregate applied to them

Counter Handling

In PromQL, counter metrics require explicit functions like rate() or increase() to calculate the rate of change. In MetoroQL, counter values are automatically presented as the difference between consecutive data points. This means:

Values represent changes between points rather than cumulative values
You usually do not need rate() or increase() just to see counter changes

For example, consider a HTTP request counter metric with one minute buckets:

# In PromQL, a query to get the number of requests 
# sent in a given minute might look something like this
sum(rate(http_requests_total{service="api"}[5m])) / 5
# or
sum(rate(http_requests_total{service="api"}[5m])) * 60

# In MetoroQL, you directly query the counter
sum(http_requests_total{service="api"})

The MetoroQL query will directly show the change in request count between data points, while in PromQL, the raw http_requests_total would show monotonically increasing cumulative values that generally aren’t immediately useful without applying rate() or increase().

Rolling Window Functions

For scalar metrics, MetoroQL also supports PromQL-style range functions when importing or writing queries that already use lookback windows:

rate(metric[5m])
irate(metric[1m])
increase(metric[10m])
avg_over_time(metric[5m])
sum_over_time(metric[5m])
min_over_time(metric[5m])
max_over_time(metric[5m])

These functions are implemented as rolling-window transforms that run before the final aggregation. For example:

avg(rate(http_requests_total[5m])) by (service)

Metoro first calculates the rolling transform for each underlying time series, then applies avg across the service grouping. Counter-based rate, irate, and increase operate on Metoro’s reset-corrected bucket deltas.

These range functions are New Relic-style rolling-window approximations. They are not exact Prometheus extrapolation or staleness semantics. If the chart bucket size is larger than the lookback, Metoro uses the bucket size as the effective window. In this version, irate uses the same rolling-rate machinery as rate.

Multi-domain Queries

One of the most powerful features of MetoroQL is the ability to query across different observability domains using special metric names:

logs - Log data
traces - Distributed tracing data
kubernetes_resources - Kubernetes resources information
Any other metric identifier is treated as a regular metric

Each of these domains has specific functions and aggregations that can be applied to them.

Forced aggregations

In mQL all timeseries queries require an explicit aggregation function such as:

sum
avg
min
max
count
histogram_quantile

For example

# The following is a valid query in promql, this will return 
# all individual timeseries (unique combination of labels and values).
container_resources_cpu_usage_seconds_total

# In mQL you must specify an aggregate
sum(container_resources_cpu_usage_seconds_total)

This behavior differs from the default PromQL behavior.

Basic Query Syntax

A simple MetoroQL time-series query has the following structure:

aggregation(metric_name{label1="value1", label2="value2"}) by (grouplabel1, grouplabel2) 

For example, to get the CPU usage of all services running in the default namespace:

max(container_resources_cpu_usage_seconds_total{namespace="default"}) by (service_name)

You can also perform arithmetic on a time series or between multiple time series. For example, to get the percent of allocated disk actually used by a service:

(sum(container_resources_disk_used_bytes) by (service_name) / 
sum(container_resources_disk_size_bytes) by (service_name)) * 100

Special Data Types

In addition to metrics, you can write mQL queries over logs, traces, and kubernetes_resources. Each of these resources has its own rules on how it can be queried.

Log Queries

Log queries support only the count aggregate.
They support all filtering and group-by operations.
Structured JSON log attributes are parsed into filterable fields.

# Count of error logs
count(logs{log_level="error"})

# Number of logs with a message matching a regex
count(logs{message=~".*failed.*"})

# Error logs by service
count(logs{log_level="error"}) by (service.name)

# Count each of the individual values of the 
# custom "caller" field for all logs that have the field.
count(logs{caller=~".+"}) by (caller)

Trace Queries

Trace queries support both the count aggregation and the trace_duration_quantile aggregation.
They support all filtering and group-by operations.
All custom attributes are queryable for filtering and group-bys.

# Count of all requests being served by currency services
count(traces{server.service.name=~".*currency.*"})

# Percent of 5XX requests served by the currency services
count(traces{http.status_code=~"5..", server.service.name=~".*currency.*"}) * 100 
/ count(traces{server.service.name=~".*currency.*"})

# P95 for the convert endpoint
trace_duration_quantile(0.95, traces{http.path="/convert",
                        server.service.name=~".*currency.*"})

Kubernetes Resource Queries

Kubernetes resource queries support both the count aggregation and all other aggregations after the json_path function is applied.
They support all filtering and group-by operations.

# Count of pods by namespace
count(kubernetes_resources{Kind="Pod"}) by (Namespace)

# Total number of replicas specified by deployments per service
sum(json_path("spec.replicas", kubernetes_resources)) by (ServiceName)

With json_path, you can:

Extract and analyze specific fields from Kubernetes resources
Use sum, avg, min, or max aggregations with the extracted values

Advanced Features

Filtering

MetoroQL supports several filtering operators:

# Exact match
metric{label="value"}

# Negation
metric{label!="value"}

# Regex matching
metric{label=~"pattern"}

# Negated regex
metric{label!~"pattern"}

Binary Operations

You can create complex queries using arithmetic operations. Supported operations are:

+ addition
- subtraction
* multiplication
/ division
% modulo
^ exponentiation
== equal
!= not equal
<= less than or equal
>= greater than or equal

# Calculate logs error rate percentage
100 * (count(logs{log_level="error"}) / sum(logs))

# Compare CPU usage across environments
sum(container_resources_cpu_usage_seconds_total{environment="production"}) - sum(container_resources_cpu_usage_seconds_total{environment="staging"})

Grouping

Group data by specific labels:

# Group by container and namespace
sum(container_memory_working_set_bytes) by (container, namespace)

# Top 5 pods by CPU usage
topk(5, max(container_cpu_usage) by (pod_name))

Get Started

AI SRE

Concepts

Traces

Logs

Metrics

Profiling

Kubernetes Resources

Dashboards

Infrastructure

Advisor

Alerts & Monitoring

Integrations

Uptime Monitoring

User Management

On-Premises

Administration

Overview

Key Differences from PromQL

Counter Handling

Rolling Window Functions

Multi-domain Queries

Forced aggregations

Basic Query Syntax

Special Data Types

Log Queries

Trace Queries

Kubernetes Resource Queries

Advanced Features

Filtering

Binary Operations

Grouping

​Overview

​Key Differences from PromQL

​Counter Handling

​Rolling Window Functions

​Multi-domain Queries

​Forced aggregations

​Basic Query Syntax

​Special Data Types

​Log Queries

​Trace Queries

​Kubernetes Resource Queries

​Advanced Features

​Filtering

​Binary Operations

​Grouping

Overview

Key Differences from PromQL

Counter Handling

Rolling Window Functions

Multi-domain Queries

Forced aggregations

Basic Query Syntax

Special Data Types

Log Queries

Trace Queries

Kubernetes Resource Queries

Advanced Features

Filtering

Binary Operations

Grouping