Distributed systems running on Kubernetes generate complex webs of inter-service communication, and when something breaks at 2 AM, you need traces that show exactly which service dropped the ball. The problem? Manually instrumenting every service with an OpenTelemetry SDK is a project unto itself, one that most platform teams never finish.
OpenTelemetry's auto-instrumentation changes the equation entirely. Using the OTel Operator for Kubernetes, you can inject telemetry collection into running workloads through pod annotations alone. The SDK integration, rebuilding of container images, and coordination with every application team to add tracing headers all become unnecessary.
This post walks through the architecture, the setup process, and the real-world gotchas you'll encounter when rolling out auto-instrumentation across a Kubernetes cluster.
Why Auto-Instrumentation Matters
Manual instrumentation gives you fine-grained control. You decide exactly which spans to create, what attributes to attach, and how context propagates across service boundaries. But it comes with significant overhead: every team must adopt the SDK, every language requires its own integration, and keeping instrumentation consistent across dozens of services is a governance nightmare.
Auto-instrumentation flips the ownership model. Instead of asking application developers to instrument their code, platform teams configure instrumentation at the infrastructure layer. The OTel Operator watches for annotated deployments and injects the appropriate instrumentation agent (as an init container for Java, Python, Node.js, and .NET, or as an eBPF sidecar for Go) without requiring changes to the application source.
The practical benefit is coverage. You can go from zero traces to full distributed tracing across an entire namespace in an afternoon. And because auto-instrumentation hooks into well-known libraries (HTTP clients, gRPC frameworks, database drivers), the spans it generates are often exactly the ones you need most for debugging production issues.
How It Works: The Architecture
The auto-instrumentation setup involves three layers working together inside your cluster.
The OTel Operator runs as a controller in a dedicated namespace. It watches for two custom resources: Instrumentation (which defines how to instrument apps) and OpenTelemetryCollector (which defines how to collect and forward telemetry). When a pod with the right annotations is created, the operator's admission webhook mutates the pod spec before it launches.
The sidecar collector runs alongside each instrumented pod. It receives telemetry from the instrumentation agent over localhost (OTLP on port 4317 or 4318) and forwards it to a central collector. This sidecar pattern keeps telemetry egress local to the pod, reducing the blast radius if the central collector is temporarily unavailable.
The central collector aggregates telemetry from all sidecars and exports it to your backend of choice, whether that's Jaeger, Grafana Tempo, Parseable, or any OTLP-compatible destination. It also handles processing like batching, attribute enrichment, and sampling.
The data flow looks like this:
App Process (with injected agent)
→ localhost:4318 (OTLP HTTP)
→ Sidecar Collector (per-pod)
→ Central Collector (cluster-wide)
→ Observability BackendFor most languages, the injection adds a single init container that copies the agent to a shared volume and sets environment variables (JAVA_TOOL_OPTIONS, NODE_OPTIONS, PYTHONPATH, or CORECLR_ENABLE_PROFILING) so the agent loads at process startup. The result is a pod with two containers: your original app (now instrumented via env vars) and the sidecar collector.
Go is the exception. Because Go compiles to static binaries with runtime hooks unavailable, auto-instrumentation uses eBPF probes attached by a privileged sidecar. This means Go-instrumented pods end up with three containers, and the setup requires a couple of extra configuration steps.
Setting It Up: Step by Step
Prerequisites
You need a Kubernetes cluster with kubectl and helm access, and cert-manager must already be deployed (the operator's admission webhooks require TLS certificates).
Install the OTel Operator
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
--namespace otel \
--create-namespace \
--set admissionWebhooks.certManager.enabled=true \
--set manager.collectorImage.repository=otel/opentelemetry-collector-contribIf you plan to instrument Go services, you must also add --set manager.autoInstrumentation.go.enabled=true because without it, the operator silently skips Go injection. This flag is specific to Go since Java, Python, Node.js, and .NET are enabled by default.
Verify the operator is running:
kubectl get pods -n otel -l app.kubernetes.io/instance=opentelemetry-operator
# Expect: 2/2 RunningCreate a Sidecar Collector
The sidecar collector is defined as an OpenTelemetryCollector custom resource with mode: sidecar. It must live in the same namespace as your application deployments.
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: sidecar-collector
namespace: my-apps
spec:
mode: sidecar
config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
send_batch_size: 10
timeout: 5s
resource:
attributes:
- action: insert
key: cluster
value: my-cluster-name
exporters:
otlp:
endpoint: central-collector.otel.svc.cluster.local:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [resource, batch]
exporters: [otlp]
metrics:
receivers: [otlp]
processors: [resource, batch]
exporters: [otlp]The resource processor is useful for stamping every span with the cluster name, making it easy to filter traces when you have multiple clusters feeding into the same backend.
Create the Instrumentation CRD
A single Instrumentation resource can define configuration for all five supported languages. The operator injects only the agent that matches the annotation on each deployment.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: auto-instrumentation
namespace: my-apps
spec:
exporter:
endpoint: http://localhost:4318
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "1"
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
python:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
dotnet:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:latest
go:
image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:latest
env:
- name: OTEL_GO_AUTO_TARGET_EXE
value: /appTwo things to note here. First, the exporter endpoint is localhost:4318, which points at the sidecar collector running in the same pod. Second, the sampler argument is set to "1" (100% sampling), which is ideal for staging. In production, you'll likely want to dial this down or use a tail-sampling strategy in the central collector.
Annotate Your Deployments
This is where the magic happens. Adding annotations to a deployment's pod template triggers injection on the next rollout.
For a Python service:
kubectl patch deployment my-python-api -n my-apps --type merge -p '{
"spec": {
"template": {
"metadata": {
"annotations": {
"sidecar.opentelemetry.io/inject": "true",
"instrumentation.opentelemetry.io/inject-python": "true"
}
}
}
}
}'For a Java service, swap inject-python for inject-java. Node.js uses inject-nodejs, .NET uses inject-dotnet.
Go requires one additional annotation specifying the absolute path to the binary inside the container:
kubectl patch deployment my-go-service -n my-apps --type merge -p '{
"spec": {
"template": {
"metadata": {
"annotations": {
"sidecar.opentelemetry.io/inject": "true",
"instrumentation.opentelemetry.io/inject-go": "true",
"instrumentation.opentelemetry.io/otel-go-auto-target-exe": "/root/my-go-service"
}
}
}
}
}'After patching, watch the pods roll out. You should see 2/2 containers for SDK-based languages or 3/3 for Go.
Gotchas That Will Bite You
After rolling this out across multiple clusters, here are the pitfalls that consistently trip people up.
The Instrumentation CRD must be in the same namespace as your pods. If you create it in the otel namespace but your apps live in my-apps, the operator silently ignores the injection annotations. There are zero error messages and it just doesn't work.
Go auto-instrumentation speaks HTTP, not gRPC. The eBPF-based Go agent exports telemetry over HTTP/1.x. If your Instrumentation CRD's exporter endpoint points to port 4317 (the gRPC port), you'll see HTTP/1.x transport connection broken: malformed HTTP response errors. Always use port 4318 for Go.
Wrong binary path means silent failure. For Go services, if the otel-go-auto-target-exe annotation points to a path that doesn't exist inside the container, the instrumentation sidecar gets stuck polling indefinitely. You'll see zero errors, zero crashes, and zero traces. Verify the path with kubectl exec before annotating.
Node.js ESM modules have limited support. If your Node.js service uses ES modules (import syntax) rather than CommonJS (require), auto-instrumentation may fail to generate spans for all libraries. This is a known limitation in the upstream OTel Node.js SDK.
Prometheus relabeling requires double-dollar escaping. If your central collector config uses $1 or $2 in relabel replacement rules, the OTel collector's config parser interprets them as environment variable references. Use $$1 and $$2 instead, or the collector will crash with cryptic errors like environment variable "2" has invalid name.
ArgoCD will overwrite your patches. If you manage deployments through GitOps, patching annotations with kubectl is a temporary fix. The next sync cycle will revert your changes. Persist annotations in your Helm values or Kustomize overlays.
Verifying the Pipeline
Once everything is deployed, validate end to end:
# 1. Pods have the expected container count
kubectl get pods -n my-apps
# 2/2 for Java/Python/Node/.NET, 3/3 for Go
# 2. Sidecar collector is healthy
kubectl logs deploy/my-service -n my-apps -c otc-container --tail=5
# "Everything is ready. Begin running and processing data."
# 3. For Go, verify eBPF agent attached successfully
kubectl logs deploy/my-service -n my-apps -c opentelemetry-auto-instrumentation --tail=5
# "instrumentation loaded successfully, starting..."
# 4. Central collector is forwarding traces
kubectl logs deploy/central-collector -n otel --tail=20 | grep -i trace
# TracesExporter lines with span countsIf traces aren't showing up in your backend, work backwards: check the central collector logs first, then the sidecar, then the instrumentation agent. The problem is almost always a misconfigured endpoint, a namespace mismatch, or a missing annotation.
Sending Traces to Your Backend
The central collector's exporter configuration determines where your telemetry lands. Any OTLP-compatible backend works. For example, to export traces over HTTP with gzip compression:
exporters:
otlphttp/traces:
compression: gzip
encoding: json
headers:
Authorization: Basic <your-credentials>
traces_endpoint: https://your-backend.example.com/v1/tracesPlatforms like Parseable can ingest OpenTelemetry traces, metrics, and logs natively over OTLP HTTP, letting you store and query all three signal types in a single backend. Whatever backend you choose, the collector's exporter pipeline makes it straightforward to switch or fan out to multiple destinations, all without changing any application code.
Where to Go from Here
Auto-instrumentation gets you broad coverage fast, but it's a starting point, and here's how to evolve the setup over time.
Start by pinning your auto-instrumentation images to specific versions rather than using latest. Version mismatches between the agent and your application's dependencies can cause subtle issues, especially with Java.
Consider moving from 100% sampling to tail-based sampling in the central collector. Capture all error traces and a percentage of successful ones. This dramatically reduces storage costs while preserving debugging capability.
Finally, treat auto-instrumentation and manual instrumentation as complementary. Auto-instrumentation gives you the distributed trace skeleton: HTTP calls, database queries, and gRPC requests. Manual instrumentation lets you add business-specific spans and attributes (user IDs, order amounts, feature flags) that make traces truly useful for understanding your system's behavior.
The beauty of the OTel Operator approach is that it's incremental. You can start with a single namespace, prove the value with one team, and expand from there, all while keeping developer code completely untouched.
