Scrape Metrics with ServiceMonitors

If your applications already expose Prometheus metrics, you can send them to Metoro with ServiceMonitor or PodMonitor resources. The Metoro Helm chart installs and configures an OpenTelemetry Collector and Target Allocator for this purpose. You do not need to run another collector or configure Prometheus remote write. The Target Allocator discovers monitor resources and distributes their scrape targets across the Metoro-managed collector replicas. The collectors scrape the targets and send the metrics through the in-cluster Metoro exporter.

The Target Allocator assigns targets; it does not scrape or forward metrics itself.

Prerequisites

Metoro is installed with the Metoro exporter Helm chart.
The ServiceMonitor and PodMonitor custom resource definitions (CRDs) are installed in the cluster.
Your workload exposes a Prometheus-compatible metrics endpoint that is reachable from the Metoro namespace.

The Metoro chart enables ServiceMonitor scraping by default, but it does not install the Prometheus Operator or the monitoring.coreos.com CRDs. If you use kube-prometheus-stack or the Prometheus Operator, you likely already have the CRDs. You do not need a running Prometheus server for Metoro to use them. Check for the CRDs:

kubectl get crd servicemonitors.monitoring.coreos.com podmonitors.monitoring.coreos.com

If either CRD is missing, install it using the Prometheus Operator installation guidance before continuing.

Verify the Metoro scraper is running

The chart creates a collector StatefulSet and a Target Allocator Deployment. If Metoro is installed in the metoro namespace, check them with:

kubectl -n metoro get statefulset,pods \
  -l app.kubernetes.io/component=service-monitor-scraper

kubectl -n metoro get deployment,pods,service \
  -l app.kubernetes.io/component=opentelemetry-targetallocator

If these resources do not exist, upgrade the Metoro Helm chart and make sure scraping is enabled in your values:

serviceMonitorScraping:
  enabled: true

Create a ServiceMonitor

The following example assumes that an application named example-api exposes Prometheus metrics on port 8080 at /metrics. First, make sure the application has a Kubernetes Service with a named port:

apiVersion: v1
kind: Service
metadata:
  name: example-api
  namespace: applications
  labels:
    app: example-api
spec:
  selector:
    app: example-api
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080

Then create a ServiceMonitor that selects the Service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-api
  namespace: applications
spec:
  selector:
    matchLabels:
      app: example-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Apply both resources:

kubectl apply -f example-api-service.yaml
kubectl apply -f example-api-servicemonitor.yaml

Three fields must line up:

ServiceMonitor.spec.selector matches labels on the Service, not labels on its Pods.
ServiceMonitor.spec.endpoints[].port matches the name of a Service port. In this example, both are metrics.
Without ServiceMonitor.spec.namespaceSelector, the ServiceMonitor finds Services only in its own namespace.

For Services in other namespaces, set spec.namespaceSelector on the ServiceMonitor:

spec:
  namespaceSelector:
    matchNames:
      - applications
  selector:
    matchLabels:
      app: example-api

See the Prometheus Operator ServiceMonitor API for options such as TLS, authentication, relabeling, and sample limits.

Verify target discovery

Allow about one minute for discovery and the first scrape. Then list the Target Allocator Service:

kubectl -n metoro get service \
  -l app.kubernetes.io/component=opentelemetry-targetallocator

Port-forward the returned service name in a separate terminal:

kubectl -n metoro port-forward service/<target-allocator-service> 8080:80

List the discovered jobs:

curl http://localhost:8080/jobs | jq

The output should contain a job similar to:

serviceMonitor/applications/example-api/0

Finally, open Metric Explorer in Metoro and search for a metric exposed by the application. The up metric can also help confirm whether the endpoint is reachable; filter it by the ServiceMonitor job or instance labels.

Limit which monitors Metoro scrapes

By default, Metoro discovers all ServiceMonitor and PodMonitor resources in all namespaces. In a shared cluster, you can restrict discovery with Helm values. For example, label the monitor resources that Metoro should use:

metadata:
  labels:
    metoro.io/scrape: "true"

Then configure both monitor selectors:

serviceMonitorScraping:
  targetAllocator:
    prometheusCR:
      serviceMonitorSelector:
        matchLabels:
          metoro.io/scrape: "true"
      podMonitorSelector:
        matchLabels:
          metoro.io/scrape: "true"

These selectors match labels on the ServiceMonitor and PodMonitor objects themselves. They do not replace the selector inside each monitor that chooses Services or Pods. You can also restrict the namespaces in which Metoro looks for monitor resources:

serviceMonitorScraping:
  targetAllocator:
    prometheusCR:
      namespaceSelector:
        matchLabels:
          metoro.io/metrics-discovery: "enabled"

Add that label to each namespace that contains monitor resources:

kubectl label namespace applications metoro.io/metrics-discovery=enabled

The chart applies this one namespaceSelector to both ServiceMonitor and PodMonitor discovery. This is different from ServiceMonitor.spec.namespaceSelector, which selects namespaces containing the target Services.

Use a PodMonitor instead

Use a PodMonitor when there is no Kubernetes Service in front of the metrics endpoint. Its selector matches Pod labels, and its endpoint port matches a named container port:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: example-worker
  namespace: applications
spec:
  selector:
    matchLabels:
      app: example-worker
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Troubleshooting

The monitor does not appear in `/jobs`

Confirm that the resource exists: kubectl get servicemonitor,podmonitor -A.
Check that labels on the monitor match the Helm serviceMonitorSelector or podMonitorSelector.
Check that the namespace containing the monitor matches the Helm namespaceSelector.

Inspect Target Allocator logs:

kubectl -n metoro logs \
  -l app.kubernetes.io/component=opentelemetry-targetallocator \
  --tail=100

The monitor appears, but it has no targets

Confirm that the monitor selector matches the Service or Pod labels.
For a ServiceMonitor, confirm that the endpoint port matches the Service port name.
For a PodMonitor, confirm that the endpoint port matches the container port name.
Check the monitor’s spec.namespaceSelector and verify that the Service or Pod exists in a selected namespace.
Confirm that the Service has ready endpoints with kubectl get endpointslices -n <namespace>.

Targets exist, but metrics do not appear in Metoro

Check NetworkPolicies, DNS, TLS, and authentication between the collector Pods in metoro and the metrics endpoint.

Inspect collector logs:

kubectl -n metoro logs \
  -l app.kubernetes.io/component=service-monitor-scraper \
  --tail=100

Check whether the same targets are also being forwarded by another Prometheus or collector configuration, which can create duplicate series.
Make sure the metric name is not reserved by Metoro.

For deeper discovery checks, see the OpenTelemetry Target Allocator troubleshooting guide.

Get Started

AI SRE

Concepts

Traces

Logs

Metrics

Profiling

Kubernetes Resources

Dashboards

Infrastructure

Advisor

Alerts & Monitoring

Integrations

Uptime Monitoring

User Management

On-Premises

Administration

Scrape Metrics with ServiceMonitors

Prerequisites

Verify the Metoro scraper is running

Create a ServiceMonitor

Verify target discovery

Limit which monitors Metoro scrapes

Use a PodMonitor instead

Troubleshooting

The monitor does not appear in `/jobs`

The monitor appears, but it has no targets

Targets exist, but metrics do not appear in Metoro

​Prerequisites

​Verify the Metoro scraper is running

​Create a ServiceMonitor

​Verify target discovery

​Limit which monitors Metoro scrapes

​Use a PodMonitor instead

​Troubleshooting

​The monitor does not appear in /jobs

​The monitor appears, but it has no targets

​Targets exist, but metrics do not appear in Metoro

Prerequisites

Verify the Metoro scraper is running

Create a ServiceMonitor

Verify target discovery

Limit which monitors Metoro scrapes

Use a PodMonitor instead

Troubleshooting

The monitor does not appear in `/jobs`

The monitor appears, but it has no targets

Targets exist, but metrics do not appear in Metoro