> ## Documentation Index
> Fetch the complete documentation index at: https://metoro.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape Metrics with ServiceMonitors

> Use ServiceMonitor and PodMonitor resources to send Prometheus metrics to Metoro

If your applications already expose Prometheus metrics, you can send them to Metoro with `ServiceMonitor` or `PodMonitor` resources. The Metoro Helm chart installs and configures an OpenTelemetry Collector and Target Allocator for this purpose. You do not need to run another collector or configure Prometheus remote write.

The Target Allocator discovers monitor resources and distributes their scrape targets across the Metoro-managed collector replicas. The collectors scrape the targets and send the metrics through the in-cluster Metoro exporter.

<Note>
  The Target Allocator assigns targets; it does not scrape or forward metrics itself.
</Note>

## Prerequisites

* Metoro is installed with the [Metoro exporter Helm chart](https://github.com/metoro-io/metoro-helm-charts).
* The `ServiceMonitor` and `PodMonitor` custom resource definitions (CRDs) are installed in the cluster.
* Your workload exposes a Prometheus-compatible metrics endpoint that is reachable from the Metoro namespace.

The Metoro chart enables ServiceMonitor scraping by default, but it does not install the Prometheus Operator or the `monitoring.coreos.com` CRDs. If you use `kube-prometheus-stack` or the Prometheus Operator, you likely already have the CRDs. You do not need a running Prometheus server for Metoro to use them.

Check for the CRDs:

```bash theme={null}
kubectl get crd servicemonitors.monitoring.coreos.com podmonitors.monitoring.coreos.com
```

If either CRD is missing, install it using the [Prometheus Operator installation guidance](https://prometheus-operator.dev/docs/getting-started/installation/) before continuing.

## Verify the Metoro scraper is running

The chart creates a collector StatefulSet and a Target Allocator Deployment. If Metoro is installed in the `metoro` namespace, check them with:

```bash theme={null}
kubectl -n metoro get statefulset,pods \
  -l app.kubernetes.io/component=service-monitor-scraper

kubectl -n metoro get deployment,pods,service \
  -l app.kubernetes.io/component=opentelemetry-targetallocator
```

If these resources do not exist, [upgrade the Metoro Helm chart](/docs/getting-started/upgrading-metoro) and make sure scraping is enabled in your values:

```yaml theme={null}
serviceMonitorScraping:
  enabled: true
```

## Create a ServiceMonitor

The following example assumes that an application named `example-api` exposes Prometheus metrics on port `8080` at `/metrics`.

First, make sure the application has a Kubernetes Service with a **named port**:

```yaml theme={null}
apiVersion: v1
kind: Service
metadata:
  name: example-api
  namespace: applications
  labels:
    app: example-api
spec:
  selector:
    app: example-api
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080
```

Then create a `ServiceMonitor` that selects the Service:

```yaml theme={null}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-api
  namespace: applications
spec:
  selector:
    matchLabels:
      app: example-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
```

Apply both resources:

```bash theme={null}
kubectl apply -f example-api-service.yaml
kubectl apply -f example-api-servicemonitor.yaml
```

Three fields must line up:

* `ServiceMonitor.spec.selector` matches labels on the **Service**, not labels on its Pods.
* `ServiceMonitor.spec.endpoints[].port` matches the **name** of a Service port. In this example, both are `metrics`.
* Without `ServiceMonitor.spec.namespaceSelector`, the ServiceMonitor finds Services only in its own namespace.

For Services in other namespaces, set `spec.namespaceSelector` on the ServiceMonitor:

```yaml theme={null}
spec:
  namespaceSelector:
    matchNames:
      - applications
  selector:
    matchLabels:
      app: example-api
```

See the [Prometheus Operator ServiceMonitor API](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1.ServiceMonitor) for options such as TLS, authentication, relabeling, and sample limits.

## Verify target discovery

Allow about one minute for discovery and the first scrape. Then list the Target Allocator Service:

```bash theme={null}
kubectl -n metoro get service \
  -l app.kubernetes.io/component=opentelemetry-targetallocator
```

Port-forward the returned service name in a separate terminal:

```bash theme={null}
kubectl -n metoro port-forward service/<target-allocator-service> 8080:80
```

List the discovered jobs:

```bash theme={null}
curl http://localhost:8080/jobs | jq
```

The output should contain a job similar to:

```text theme={null}
serviceMonitor/applications/example-api/0
```

Finally, open **Metric Explorer** in Metoro and search for a metric exposed by the application. The `up` metric can also help confirm whether the endpoint is reachable; filter it by the ServiceMonitor job or instance labels.

## Limit which monitors Metoro scrapes

By default, Metoro discovers all ServiceMonitor and PodMonitor resources in all namespaces. In a shared cluster, you can restrict discovery with Helm values.

For example, label the monitor resources that Metoro should use:

```yaml theme={null}
metadata:
  labels:
    metoro.io/scrape: "true"
```

Then configure both monitor selectors:

```yaml theme={null}
serviceMonitorScraping:
  targetAllocator:
    prometheusCR:
      serviceMonitorSelector:
        matchLabels:
          metoro.io/scrape: "true"
      podMonitorSelector:
        matchLabels:
          metoro.io/scrape: "true"
```

These selectors match labels on the `ServiceMonitor` and `PodMonitor` objects themselves. They do not replace the selector inside each monitor that chooses Services or Pods.

You can also restrict the namespaces in which Metoro looks for monitor resources:

```yaml theme={null}
serviceMonitorScraping:
  targetAllocator:
    prometheusCR:
      namespaceSelector:
        matchLabels:
          metoro.io/metrics-discovery: "enabled"
```

Add that label to each namespace that contains monitor resources:

```bash theme={null}
kubectl label namespace applications metoro.io/metrics-discovery=enabled
```

The chart applies this one `namespaceSelector` to both ServiceMonitor and PodMonitor discovery. This is different from `ServiceMonitor.spec.namespaceSelector`, which selects namespaces containing the target Services.

## Use a PodMonitor instead

Use a `PodMonitor` when there is no Kubernetes Service in front of the metrics endpoint. Its selector matches Pod labels, and its endpoint port matches a named container port:

```yaml theme={null}
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: example-worker
  namespace: applications
spec:
  selector:
    matchLabels:
      app: example-worker
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 30s
```

## Troubleshooting

### The monitor does not appear in `/jobs`

* Confirm that the resource exists: `kubectl get servicemonitor,podmonitor -A`.
* Check that labels on the monitor match the Helm `serviceMonitorSelector` or `podMonitorSelector`.
* Check that the namespace containing the monitor matches the Helm `namespaceSelector`.
* Inspect Target Allocator logs:

  ```bash theme={null}
  kubectl -n metoro logs \
    -l app.kubernetes.io/component=opentelemetry-targetallocator \
    --tail=100
  ```

### The monitor appears, but it has no targets

* Confirm that the monitor selector matches the Service or Pod labels.
* For a ServiceMonitor, confirm that the endpoint port matches the Service port **name**.
* For a PodMonitor, confirm that the endpoint port matches the container port **name**.
* Check the monitor's `spec.namespaceSelector` and verify that the Service or Pod exists in a selected namespace.
* Confirm that the Service has ready endpoints with `kubectl get endpointslices -n <namespace>`.

### Targets exist, but metrics do not appear in Metoro

* Check NetworkPolicies, DNS, TLS, and authentication between the collector Pods in `metoro` and the metrics endpoint.

* Inspect collector logs:

  ```bash theme={null}
  kubectl -n metoro logs \
    -l app.kubernetes.io/component=service-monitor-scraper \
    --tail=100
  ```

* Check whether the same targets are also being forwarded by another Prometheus or collector configuration, which can create duplicate series.

* Make sure the metric name is not [reserved by Metoro](/docs/metrics/custom-metrics#reserved-names).

For deeper discovery checks, see the [OpenTelemetry Target Allocator troubleshooting guide](https://opentelemetry.io/docs/platforms/kubernetes/operator/troubleshooting/target-allocator/).