Kubernetes observability with OpenTelemetry: deploying the Collector and instrumenting workloads

Table of contents

• What you will learn
• Prerequisites
• What OpenTelemetry adds to a Kubernetes observability stack
• Architecture: signals, Collector, Operator
• What OpenTelemetry is not
• Install cert-manager and the OpenTelemetry Operator
• Deploy a DaemonSet Collector for node-level signals
• Deploy a Deployment Collector for cluster-level signals
• Auto-instrument workloads with the Instrumentation CRD
• Route signals to backends
• Verify the pipeline end-to-end
• Common gotchas
• What you learned
• Where to go next

What you will learn

By the end of this tutorial you will have the OpenTelemetry Operator running on your cluster, two Collectors collecting the right signals at the right scope, and at least one sample workload producing traces without a line of application code changing. You will understand why there are two Collectors instead of one, which receivers belong where, and how to route the output to a Prometheus, Grafana Tempo, or Jaeger backend.

Prerequisites

Before starting, make sure you have:

• A running Kubernetes cluster, version 1.25 or later. A managed cluster (GKE, EKS, AKS) or a local kind/minikube cluster both work. This tutorial was written against OpenTelemetry Operator v0.148.0, released in March 2026.
• Helm 3.9 or later configured to talk to your cluster. The Operator Helm chart requires Helm 3.9+.
• kubectl configured with cluster-admin privileges. The Operator installs CRDs and cluster-wide resources.
• A way to reach an observability backend. This tutorial uses Prometheus and Grafana Tempo as examples; if you already have Prometheus running via kube-prometheus-stack, you can reuse that and only add Tempo for traces.
• Familiarity with Kubernetes Deployments, DaemonSets, and pod annotations. If TLS automation is new to you, the cert-manager tutorial covers the prerequisite cleanly.

What OpenTelemetry adds to a Kubernetes observability stack

If you already have Prometheus scraping metrics and Fluent Bit shipping logs, the honest question is what OpenTelemetry adds. The answer has three parts.

A third signal you probably do not have: distributed traces. Metrics tell you that the checkout API is slow. Logs tell you what the checkout API printed while it was slow. Traces show you the full request path across the five services the checkout API called, where the time went in each hop, and which hop caused the tail latency. Prometheus does not produce traces. Fluent Bit does not produce traces. Without traces, diagnosing latency in a service mesh is guesswork.

A single wire format. OpenTelemetry defines the OTLP protocol, a unified encoding for traces, metrics, and logs over gRPC (port 4317) and HTTP (port 4318). Every major backend now speaks OTLP natively: Prometheus via OTLP ingestion, Grafana Tempo, Jaeger, Elasticsearch APM, Datadog, Honeycomb, New Relic. You pick the backend. You never re-instrument when you switch.

Auto-instrumentation without code changes. The OpenTelemetry Operator injects an init-container into annotated pods that bootstraps the language runtime with an OpenTelemetry agent before your application starts. Java, Python, Node.js, .NET, Go, Apache HTTPD, and Nginx are supported. You add one annotation. Traces appear.

This is why OpenTelemetry is worth adding even when you already have a working Prometheus setup. It fills a gap rather than replacing what works.

Architecture: signals, Collector, Operator

The architecture has three layers.

Signals. Traces, metrics, and logs. These are the three telemetry signals OpenTelemetry defines. Your application emits them; Collectors receive, transform, and export them.

Collectors. A Collector is a standalone process that receives telemetry, runs it through a pipeline of processors, and exports it to one or more backends. Collectors are horizontally scalable and backend-agnostic. On Kubernetes, the official guidance is to run two Collectors:

• A DaemonSet Collector on every node to collect node-local telemetry (kubelet stats, container logs, host metrics) and to receive OTLP from workloads on the same node. "A daemonset is used to guarantee that this instance of the collector is installed on all nodes."
• A Deployment Collector with a single replica for cluster-wide telemetry (cluster-scoped metrics, Kubernetes events). "A deployment with exactly one replica ensures that we don't produce duplicate data."

Operator. The OpenTelemetry Operator is a Kubernetes operator that reconciles two CRDs: OpenTelemetryCollector (which defines a Collector deployment) and Instrumentation (which defines how to auto-instrument pods). The Operator also injects auto-instrumentation init-containers into pods that carry the right annotation.

You could run the Collector without the Operator by applying raw manifests. You lose the CRD-driven workflow, the auto-instrumentation injector, and the target allocator for Prometheus-based metrics. For a production cluster, the Operator is worth the extra component.

What OpenTelemetry is not

Three misconceptions come up consistently.

OpenTelemetry does not replace Prometheus. OpenTelemetry defines signals, SDKs, and a wire format. It does not include a time-series database, a query language like PromQL, or an alerting engine. You still need a metrics backend. Prometheus remains an excellent one; the OTLP receiver in Prometheus 3.x accepts metrics directly from Collectors. OpenTelemetry sits in front of Prometheus, not in place of it.

Auto-instrumentation does not require source code changes. This is the whole point. The Operator's automatic instrumentation injects an init-container named opentelemetry-auto-instrumentation into any pod carrying the right annotation. That init-container copies the language agent into a shared volume; your app container loads it via environment variables the Operator sets. Your Dockerfile, your source code, your CI pipeline: all untouched.

One Collector deployment is not sufficient for production. A single Deployment Collector cannot collect kubelet stats from every node (it is only scheduled on one) and cannot tail /var/log/pods/*/*/*.log across the cluster. A single DaemonSet Collector would duplicate cluster-scoped data (one copy per node) when scraping the Kubernetes API. The DaemonSet + Deployment split is not an optimization. It is a correctness requirement.

Install cert-manager and the OpenTelemetry Operator

The Operator uses validating and mutating admission webhooks, which require TLS. The Helm chart supports three ways to provide that TLS: cert-manager (recommended), auto-generated self-signed certs from the chart itself, or certs you supply. This tutorial uses cert-manager because it is the path that scales to other operators you will add later.

Step 1: install cert-manager

# cert-manager v1.20.2 via OCI registry (recommended by jetstack)
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.20.2 \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

The cert-manager installation docs cover legacy HTTP repository mode if your environment blocks OCI registries.

Expected output. helm list -n cert-manager shows cert-manager in deployed state, and kubectl get pods -n cert-manager shows three pods running: the controller, webhook, and cainjector.

Step 2: add the OpenTelemetry Helm repository

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Step 3: install the Operator

# Operator v0.148.0; overrides the default collector image to the
# Kubernetes-aware distribution (otelcol-k8s) which bundles the
# k8sattributes, k8scluster, kubeletstats and filelog components.
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
  --namespace opentelemetry \
  --create-namespace \
  --set "manager.collectorImage.repository=ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s" \
  --set admissionWebhooks.certManager.enabled=true

The manager.collectorImage.repository override matters: the default collector image is the minimal otelcol distribution and lacks Kubernetes-specific receivers. The opentelemetry-collector-k8s distribution bundles the ones you actually need.

Expected output. Within about a minute:

kubectl get pods -n opentelemetry

Shows one pod named opentelemetry-operator-<hash> in Running state. kubectl get crd | grep opentelemetry shows opentelemetrycollectors.opentelemetry.io and instrumentations.opentelemetry.io as installed CRDs.

If the Operator pod is stuck in ContainerCreating with a webhook-related error, cert-manager has not issued the cert yet. Wait 30 seconds and check kubectl get certificate -n opentelemetry; once the certificate is Ready=True, the webhook starts.

Deploy a DaemonSet Collector for node-level signals

The DaemonSet Collector runs on every node and handles four responsibilities:

1. Receive OTLP from local workloads on ports 4317 (gRPC) and 4318 (HTTP). Keeping this local-only avoids unnecessary network hops.
2. Collect kubelet stats (pod and container CPU/memory) via the kubeletstats receiver.
3. Tail container logs from /var/log/pods/*/*/*.log via the filelog receiver.
4. Enrich every signal with Kubernetes metadata (pod name, namespace, labels) via the k8sattributes processor.

Apply the following manifest:

# otel-collector-daemonset.yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-agent
  namespace: opentelemetry
spec:
  mode: daemonset
  # The k8s distribution ships with the receivers this config needs.
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s:0.120.0
  # Host-level mounts for log tailing and kubelet stats.
  volumeMounts:
    - name: varlogpods
      mountPath: /var/log/pods
      readOnly: true
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
  volumes:
    - name: varlogpods
      hostPath:
        path: /var/log/pods
    - name: varlibdockercontainers
      hostPath:
        path: /var/lib/docker/containers
  # Must run on every node, including control plane.
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
  # Needed so kubeletstats can authenticate to the local kubelet.
  env:
    - name: K8S_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
  config:
    receivers:
      # Applications send OTLP to the local DaemonSet via localhost.
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      # Pod/container metrics from the local kubelet.
      kubeletstats:
        auth_type: serviceAccount
        collection_interval: 30s
        endpoint: https://${env:K8S_NODE_NAME}:10250
        insecure_skip_verify: true     # acceptable for kubelet on-node
      # Container stdout/stderr logs.
      filelog:
        include:
          - /var/log/pods/*/*/*.log
        exclude:
          - /var/log/pods/opentelemetry_*/*/*.log   # avoid log loops
        start_at: end
        include_file_path: true

    processors:
      # Enriches signals with pod/namespace/label metadata.
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.node.name
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.uid
          - sources:
              - from: connection
      batch: {}                                     # always batch before export

    exporters:
      # Forward everything to the cluster Collector (see next section).
      otlp:
        endpoint: otel-gateway.opentelemetry.svc.cluster.local:4317
        tls:
          insecure: true                            # in-cluster traffic

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8sattributes, batch]
          exporters: [otlp]
        metrics:
          receivers: [otlp, kubeletstats]
          processors: [k8sattributes, batch]
          exporters: [otlp]
        logs:
          receivers: [otlp, filelog]
          processors: [k8sattributes, batch]
          exporters: [otlp]

kubectl apply -f otel-collector-daemonset.yaml

Expected output. kubectl get daemonset -n opentelemetry shows otel-agent-collector with DESIRED equal to the number of nodes in the cluster and the same value in READY. Each node now has a Collector pod listening on the host network for OTLP on 4317/4318 and scraping its local kubelet.

If the DaemonSet is healthy but kubeletstats shows errors in the logs about TLS, the insecure_skip_verify: true flag is the pragmatic fix. For a strict production setup, mount the kubelet CA bundle instead; the kubeletstats receiver docs show both paths.

Deploy a Deployment Collector for cluster-level signals

The Deployment Collector runs with exactly one replica and handles signals that only make sense at cluster scope:

1. Cluster-wide metrics (node conditions, pod phases, deployment replicas) via the k8scluster receiver.
2. Kubernetes events via the k8sobjects receiver.
3. Receive and forward telemetry from the DaemonSet Collectors on to the backends.

# otel-collector-deployment.yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-gateway
  namespace: opentelemetry
spec:
  mode: deployment
  replicas: 1                         # cluster-scoped receivers require exactly one
  image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s:0.120.0
  config:
    receivers:
      # DaemonSets push OTLP here.
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

      # Cluster-wide metrics (pod phases, node conditions, deployment status).
      k8s_cluster:
        auth_type: serviceAccount
        collection_interval: 30s
        node_conditions_to_report: [Ready, MemoryPressure, DiskPressure, PIDPressure]

      # Kubernetes events as log records.
      k8sobjects:
        auth_type: serviceAccount
        objects:
          - name: events
            mode: watch
            group: events.k8s.io

    processors:
      batch: {}

    exporters:
      # Traces -> Tempo.
      otlp/tempo:
        endpoint: tempo.observability.svc.cluster.local:4317
        tls:
          insecure: true
      # Metrics -> Prometheus via OTLP ingestion (Prometheus 3.x).
      otlphttp/prometheus:
        endpoint: http://kube-prom-stack-kube-prom-prometheus.monitoring.svc.cluster.local:9090/api/v1/otlp
        tls:
          insecure: true
      # Logs -> a Loki OTLP endpoint, or swap for your log backend.
      otlphttp/loki:
        endpoint: http://loki-gateway.logging.svc.cluster.local/otlp
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp/tempo]
        metrics:
          receivers: [otlp, k8s_cluster]
          processors: [batch]
          exporters: [otlphttp/prometheus]
        logs:
          receivers: [otlp, k8sobjects]
          processors: [batch]
          exporters: [otlphttp/loki]

kubectl apply -f otel-collector-deployment.yaml

Expected output. kubectl get deployment -n opentelemetry shows otel-gateway-collector with READY 1/1. The Operator has also created a Service named otel-gateway-collector.opentelemetry.svc.cluster.local on ports 4317 and 4318, which is exactly what the DaemonSet's OTLP exporter targets.

Auto-instrument workloads with the Instrumentation CRD

The Operator's auto-instrumentation feature injects a language agent into pods that carry the right annotation. You configure it once per namespace with an Instrumentation CR.

Step 1: create an Instrumentation resource

# instrumentation.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
  namespace: production
spec:
  # Traces and metrics from instrumented apps go here.
  # Points at the local DaemonSet Collector on the same node.
  exporter:
    endpoint: http://$(K8S_NODE_IP):4318
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"                  # sample 25% of root traces
  # Language defaults; override per-language if needed.
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  dotnet:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:latest

kubectl apply -f instrumentation.yaml

The $(K8S_NODE_IP) placeholder is substituted by the Operator at pod start with the node IP, so each pod sends OTLP to the DaemonSet Collector on its own node. That avoids cross-node traffic for high-throughput signals.

Step 2: annotate your workloads

Add a language-specific annotation to the pod template. For a Java Deployment:

# order-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
      annotations:
        # One annotation. That's the entire instrumentation change.
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
        - name: app
          image: registry.example.com/order-api:1.14.0
          ports:
            - containerPort: 8080

The annotation values for other runtimes:

• Python: instrumentation.opentelemetry.io/inject-python: "true"
• Node.js: instrumentation.opentelemetry.io/inject-nodejs: "true"
• .NET: instrumentation.opentelemetry.io/inject-dotnet: "true"
• Go: instrumentation.opentelemetry.io/inject-go: "true" (eBPF-based, still work in progress as of Operator v0.148.0; requires elevated permissions)
• Apache HTTPD: instrumentation.opentelemetry.io/inject-apache-httpd: "true"
• Nginx: instrumentation.opentelemetry.io/inject-nginx: "true"

Expected behavior. On the next rollout of order-api, kubectl describe pod on a new pod shows an init-container named opentelemetry-auto-instrumentation-java and environment variables like OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME=order-api, and JAVA_TOOL_OPTIONS=-javaagent:/otel-auto-instrumentation-java/javaagent.jar. The application container is otherwise unchanged.

A note on Go

Go auto-instrumentation is fundamentally different. The Go compiler produces a static binary with no runtime agent hook, so the Operator injects an eBPF-based sidecar that traces the binary from the kernel. That sidecar needs privileged: true and runAsUser: 0, and you have to set instrumentation.opentelemetry.io/otel-go-auto-target-exe to the absolute path of your Go binary inside the container. For production Go workloads, adding the OpenTelemetry SDK to the source is usually the cleaner path.

Route signals to backends

The Deployment Collector example above already shows three common routes. A quick reference:

Traces to Grafana Tempo. Install Tempo via its Helm chart, then point otlp/tempo at its OTLP ingestion endpoint on port 4317. Tempo stores traces in object storage and queries via TraceQL in Grafana.

Traces to Jaeger. Jaeger has native OTLP support since version 1.35. Swap the Tempo exporter for:

otlp/jaeger:
  endpoint: jaeger-collector.observability.svc.cluster.local:4317
  tls:
    insecure: true

Metrics to Prometheus. Prometheus 3.x accepts OTLP metrics directly at /api/v1/otlp/v1/metrics. Enable the feature via --web.enable-otlp-receiver on the Prometheus pod; if you use kube-prometheus-stack, set prometheus.prometheusSpec.enableFeatures: [otlp-write-receiver] in the chart values.

Metrics to an existing Prometheus-only backend. Use the prometheusremotewrite exporter instead of otlphttp. It speaks the legacy Prometheus remote-write protocol.

Logs to Loki. Loki 3.0+ accepts OTLP at /otlp/v1/logs. Point the otlphttp/loki exporter at the Loki gateway.

Logs to Elasticsearch. Use the elasticsearch exporter (in the otelcol-contrib distribution, not otelcol-k8s). If Elasticsearch is already your log backend via Fluent Bit and the EFK stack, keeping Fluent Bit for logs and using OpenTelemetry only for traces and metrics is a valid split.

Verify the pipeline end-to-end

Before declaring victory, walk signals through the full pipeline.

Check the Collector logs

# DaemonSet side
kubectl logs -n opentelemetry -l app.kubernetes.io/name=otel-agent-collector --tail=50

# Deployment side
kubectl logs -n opentelemetry -l app.kubernetes.io/name=otel-gateway-collector --tail=50

Look for lines containing Everything is ready. Begin running and processing data. A connection refused error on the DaemonSet side points at the Deployment Collector Service name; a TLS error on the backend exporter points at certificate trust.

Generate a trace

Deploy any annotated workload and hit an HTTP endpoint on it. For a quick smoke test, the OpenTelemetry demo application ships with a microservices stack that emits traces, metrics, and logs in realistic patterns.

Confirm traces reach Tempo

In Grafana, select the Tempo datasource and search for spans with service.name = order-api (or whatever service name the Operator set). You should see traces appear within 30 seconds.

Confirm metrics reach Prometheus

In the Prometheus UI (Status > Targets is not relevant here because this is push, not pull), run the query {__name__=~".+", job="otel-collector"}. You should see a non-zero result count.

Confirm logs reach Loki

In Grafana Explore with the Loki datasource, query {k8s_namespace_name="production"}. Logs with enriched Kubernetes metadata appear.

Checkpoint. If all three signals land in their respective backends and carry pod name, namespace, and node labels as resource attributes, the pipeline is working. The k8sattributes processor is the component doing that enrichment; if labels are missing, verify its ClusterRole has permission to list pods.

Common gotchas

The default collector image is not the k8s distribution. The plain otelcol image does not include k8sattributes, k8scluster, kubeletstats, or filelog. Always override manager.collectorImage.repository to the opentelemetry-collector-k8s image when installing the Operator.

Two Collectors producing duplicate data. If you see each cluster-scoped metric twice, the k8s_cluster receiver is running in both the DaemonSet and Deployment Collectors. Move it to the Deployment Collector only, and keep only node-local receivers in the DaemonSet.

Webhook errors after Operator install. If OpenTelemetryCollector apply fails with failed to call webhook: x509 certificate signed by unknown authority, cert-manager has not yet issued the webhook cert. Check kubectl get certificate -n opentelemetry -o wide and wait for Ready=True. Installing the Operator before cert-manager has fully come up is the usual cause.

Log loops from the filelog receiver. The DaemonSet Collector tails /var/log/pods/*/*/*.log, which includes the Collector's own logs. Without the exclude pattern in the filelog receiver config, every Collector log line becomes a log record that the Collector processes and exports, which produces more log lines. The opentelemetry-collector Helm chart documentation calls this out explicitly.

Auto-instrumentation annotation on the Deployment spec, not the pod template. The injector watches pod metadata, not Deployment metadata. Put the annotation under spec.template.metadata.annotations, not under the Deployment's own metadata.annotations.

Mixing otlp and otlphttp exporter names. They are different exporters with different config shapes. otlp is gRPC; otlphttp is HTTP/protobuf. Using an HTTP endpoint in otlp (or vice versa) produces confusing connection errors.

What you learned

You now have the OpenTelemetry Operator reconciling Collectors on your cluster. A DaemonSet Collector on every node handles node-local signals and receives OTLP from same-node workloads. A Deployment Collector with one replica collects cluster-scoped metrics and events, and forwards everything to the backends of your choice. Auto-instrumentation injects a language agent into pods through a single annotation, no source-code change required. Traces, metrics, and logs land in Tempo, Prometheus, and Loki with full Kubernetes metadata attached.

You also know where the sharp edges are: the collector image override, the DaemonSet vs Deployment split as a correctness requirement rather than an optimization, the cert-manager dependency, and the log-loop trap in the filelog receiver.

Where to go next

• Add PrometheusRule alerts on the metrics you now collect. The Prometheus tutorial covers rule authoring and Alertmanager routing.
• If your log volume justifies it, keep Fluent Bit for stdout logs and use OpenTelemetry only for traces and metrics. The Fluent Bit tutorial walks through that setup.
• Harden the Operator's TLS with a real cert-manager ClusterIssuer instead of the self-signed default; the cert-manager tutorial has the pattern.