OpenTelemetry Collector on Kubernetes: agent vs gateway

An OpenTelemetry Collector works almost anywhere in a Kubernetes cluster. That freedom is useful until you have to decide where it should run.

A DaemonSet puts one Collector on every node. A Deployment gives the cluster a shared Collector service. Both receive, process and export telemetry, but they see different traffic and fail in different ways.

Use an agent when collection depends on the node. Use a gateway when processing depends on telemetry from many workloads. Run both when you need local collection and a controlled central path to the backend.

That is the short answer. The rest of the design comes from deciding which work belongs at each layer.

Agent vs gateway at a glance

An agent and a gateway are roles, not separate Collector binaries. The same distribution can do either job. Placement, receivers and pipeline configuration create the role.

If all telemetry already arrives over OTLP and no node-local receiver is required, a gateway may be enough. If you only collect container logs and node metrics, agents can export directly to the backend. Larger installations often use both.

What agent mode means in Kubernetes

An agent Collector runs close to the workload that produces the data. In Kubernetes, this normally means a DaemonSet with one Collector pod on every eligible node.

The agent can mount node paths and talk to node-local endpoints. That makes it the natural home for:

• container logs under /var/log/pods
• kubelet and host metrics
• resource detection tied to the node
• Kubernetes metadata enrichment
• a nearby OTLP endpoint for applications on that node
• early filtering or redaction before data leaves the node

The Kubernetes logs with the OpenTelemetry Collector guide uses this pattern because a central Deployment cannot read every node's log files through one local mount.

Agents also spread collection load across the cluster. Adding a node adds another Collector. A busy node affects its own agent before it affects collectors on other nodes.

There is a cost. Every node now runs and upgrades a Collector. Heavy transforms steal CPU from application workloads and direct export places backend credentials on every agent. A backend outage can also create a queue on every node.

Keep the agent pipeline small. Collect local data, add metadata that depends on local context, remove data that must not leave the node, batch it and forward it.

What gateway mode means in Kubernetes

A gateway is a shared Collector endpoint. Applications or agents send OTLP to a Kubernetes Service, which distributes traffic across Collector replicas running as a Deployment or StatefulSet.

The gateway is a better home for work that benefits from a cluster-wide view:

• routing telemetry by tenant, team or environment
• keeping backend credentials in one tier
• applying shared filters and transforms
• maintaining larger retry queues
• exporting to several backends
• tail sampling complete traces
• controlling egress from the cluster

Gateways scale independently from nodes. Ten new worker nodes do not require ten new gateways. You add gateway replicas when telemetry rate, processor load or queue pressure calls for them.

A gateway is not automatically highly available. One Deployment replica is still one failure point. Run more than one replica, distribute traffic and watch Collector queue, refusal and export metrics. The OTel Collector metrics guide covers those signals.

Stateful processing needs extra care. Tail sampling must see every span from the same trace. Round-robin load balancing can split a trace across gateway replicas and produce incomplete decisions. Route spans by trace ID with the load-balancing exporter before they reach the sampling tier.

Metrics have a related constraint. Stateful metric processors and receivers must preserve the OpenTelemetry single-writer principle. More replicas do not fix a pipeline that sends the same cumulative series through several writers.

Three deployment patterns

Agent only

workloads -> node agent -> backend

Use this for node-local collection when the backend already provides a stable OTLP endpoint and distributing credentials is acceptable.

It has the shortest path and no gateway tier to operate. It becomes awkward when every agent needs large queues, complex routing or several destination credentials.

Gateway only

workloads -> Kubernetes Service -> gateway replicas -> backend

Use this when applications already emit OTLP and the pipeline does not need local files, host metrics or other node-scoped inputs.

The gateway gives applications one endpoint and centralizes policy. The trade-off is a larger shared failure domain. If the service, network path or gateway fleet fails, telemetry from the cluster loses its route.

Agent and gateway

workloads -> node agents -> gateway replicas -> backend

Use both when local collection and central control matter. Agents handle node context. Gateways handle shared processing, credentials, queues and export.

This is not a rule for every cluster. It adds another network hop, another queue boundary and another Collector configuration. Those costs are reasonable when the gateway removes heavier work and sensitive credentials from every node.

Put each job in the right layer

The combined pattern works when the two configurations have distinct responsibilities.

The Kubernetes attributes processor can run in either layer. On agents, restrict discovery to pods on the same node so every DaemonSet instance does not watch the whole cluster. On a gateway, make sure incoming telemetry still carries attributes or connection information that can be associated with a pod. Enriching at the agent is usually easier for node-local data.

The memory limiter guide explains why memory_limiter belongs near the start of both pipelines. Redaction should also happen before export; the PII removal guide shows the processors used for that boundary.

Install the gateway with Helm

The official OpenTelemetry Collector Helm chart supports daemonset, deployment and statefulset modes. Install the gateway first so agents have a destination when they start.

Add the chart repository:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Create the Parseable credentials in the observability namespace:

kubectl create namespace observability
 
kubectl create secret generic parseable-otel \
  --namespace observability \
  --from-literal=url='https://your-parseable-ingestor.example.com' \
  --from-literal=api-key='your-ingestion-api-key' \
  --from-literal=dataset='kubernetes-otel'

Save this as gateway-values.yaml:

fullnameOverride: otel-gateway
mode: deployment
replicaCount: 2
 
image:
  repository: otel/opentelemetry-collector-contrib
 
extraEnvs:
  - name: PARSEABLE_URL
    valueFrom:
      secretKeyRef:
        name: parseable-otel
        key: url
  - name: PARSEABLE_API_KEY
    valueFrom:
      secretKeyRef:
        name: parseable-otel
        key: api-key
  - name: PARSEABLE_DATASET
    valueFrom:
      secretKeyRef:
        name: parseable-otel
        key: dataset
 
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
 
  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 75
      spike_limit_percentage: 15
    batch:
      timeout: 5s
      send_batch_size: 1024
 
  exporters:
    otlp_http/parseable:
      endpoint: ${env:PARSEABLE_URL}
      encoding: json
      headers:
        X-API-Key: ${env:PARSEABLE_API_KEY}
        X-P-Stream: ${env:PARSEABLE_DATASET}
        X-P-Log-Source: otel
 
  extensions:
    health_check:
      endpoint: 0.0.0.0:13133
 
  service:
    extensions: [health_check]
    pipelines:
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp_http/parseable]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp_http/parseable]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp_http/parseable]

Install it:

helm upgrade --install otel-gateway \
  open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values gateway-values.yaml

The endpoint is now stable inside the cluster:

otel-gateway.observability.svc.cluster.local:4317

The fullnameOverride makes that service name predictable. Without it, use the service name created by your Helm release.

Install agents as a DaemonSet

Save this as agent-values.yaml:

fullnameOverride: otel-agent
mode: daemonset
 
image:
  repository: otel/opentelemetry-collector-contrib
 
presets:
  logsCollection:
    enabled: true
    includeCollectorLogs: false
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true
 
config:
  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 75
      spike_limit_percentage: 15
    batch:
      timeout: 5s
      send_batch_size: 512
 
  exporters:
    otlp/gateway:
      endpoint: otel-gateway.observability.svc.cluster.local:4317
      tls:
        insecure: true
 
  service:
    pipelines:
      logs:
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlp/gateway]
      metrics:
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlp/gateway]
      traces:
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlp/gateway]

Install the DaemonSet:

helm upgrade --install otel-agent \
  open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values agent-values.yaml

The chart presets add receivers, mounts, permissions and processor configuration needed for their data sources. Read the rendered manifest before production rollout instead of treating a preset as a permanent black box:

helm template otel-agent \
  open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values agent-values.yaml > rendered-agent.yaml

The Collector configuration guide explains how Helm's merged configuration becomes active pipelines.

Scale without breaking signal semantics

Agents and gateways scale differently.

An agent DaemonSet follows node count. Give each pod realistic CPU and memory requests, then watch whether one busy node produces refusals or sustained queue growth. Adding more agent pods to the same node rarely helps because the receivers still compete for the same local sources.

A stateless gateway can scale horizontally behind a Kubernetes Service. Start with at least two replicas when the gateway is part of the production export path. Scale on sustained CPU, memory and Collector throughput or queue signals rather than CPU alone.

Do not scale stateful processors as though every replica were independent:

• Tail sampling needs trace affinity.
• Delta conversion needs consistent metric streams.
• Cluster-wide receivers can duplicate data when every replica scrapes the same source.
• Persistent queues need storage that survives pod replacement if that durability is required.

Split stateful work into a dedicated tier when its routing or scaling rules differ from ordinary OTLP forwarding.

Verify the path

Check the workloads first:

kubectl get daemonset,deployment,pods \
  --namespace observability \
  -l app.kubernetes.io/name=opentelemetry-collector

Then inspect both Collector layers:

kubectl logs --namespace observability daemonset/otel-agent --tail=100
kubectl logs --namespace observability deployment/otel-gateway --tail=100

Look for connection errors from agents to otel-gateway:4317, authentication failures from the gateway to Parseable, refused telemetry and exporter queue errors.

Send a controlled application request and confirm that its telemetry contains k8s.cluster.name, k8s.namespace.name, k8s.pod.name and service.name. The semantic conventions guide explains why those stable resource attributes are more useful than labels invented by each team.

Finally, stop one gateway pod. The Deployment should replace it, agents should continue exporting through the Service and queues should remain bounded. A successful pod status alone does not prove the telemetry path works.

Common deployment mistakes

Running filelog on a gateway

A gateway pod only sees files mounted from its own node. Use a DaemonSet for container log paths.

Doing every transform on agents

Agents share nodes with applications. Keep expensive shared processing in gateways unless the data must be changed before leaving the node.

Putting one gateway behind a Service

The Service gives a stable name, not redundancy. Use several replicas and test a pod failure.

Scaling tail sampling with round robin

All spans for one trace must reach the same sampling decision. Use trace-aware load balancing.

Running cluster receivers on every replica

Receivers that observe the whole cluster can emit duplicate telemetry. Give them one active collector or a coordination mechanism designed for the receiver.

Losing Kubernetes metadata before enrichment

Once agents forward telemetry through a gateway, the gateway may see the agent connection rather than the original pod. Preserve resource attributes or enrich at the agent.

A deployment checklist

• Use a DaemonSet for node-local logs and metrics.
• Use a Deployment for a shared OTLP gateway.
• Keep agent processing small.
• Keep backend credentials at the gateway when possible.
• Protect both layers with memory limits and bounded queues.
• Run more than one production gateway replica.
• Preserve trace affinity before tail sampling.
• Avoid duplicate cluster-wide receivers.
• Monitor both Collector layers independently.
• Test agent, gateway and backend failures before rollout.