OpenTelemetry Collector

Applications and infrastructure send telemetry to the OpenTelemetry Collector. The Collector processes the data and exports it to a backend. Run it between your workloads and Parseable to manage batching, retries, memory protection, routing, and credentials.

Configure one Collector to send logs, metrics, and traces to separate Parseable datasets over OTLP HTTP.

Architecture

Applications and infrastructure
  -> OTLP gRPC :4317 or OTLP HTTP :4318
  -> OpenTelemetry Collector
       -> memory limiter
       -> batch processor
       -> sending queue and retries
  -> Parseable
       -> otel-logs
       -> otel-metrics
       -> otel-traces

The Collector builds and runs telemetry pipelines. Parseable stores and queries their output.

Prerequisites

• A running Parseable deployment
• An OpenTelemetry Collector installation containing the components used by your receivers
• A Parseable API key with write access to the target datasets
• Network access from the Collector to the Parseable endpoint

Use the OpenTelemetry Collector Contrib distribution when you need components such as filelog, Prometheus, Kubernetes receivers, or Kubernetes metadata processors.

Configure the Collector

This configuration accepts all three signals over OTLP. Separate Parseable exporters route each signal to its own dataset.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlphttp/parseable_logs:
    endpoint: ${env:PARSEABLE_ENDPOINT}
    encoding: proto
    headers:
      X-API-Key: ${env:PARSEABLE_API_KEY}
      X-P-Stream: otel-logs
      X-P-Log-Source: otel-logs
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0s
    sending_queue:
      enabled: true
      queue_size: 10000

  otlphttp/parseable_metrics:
    endpoint: ${env:PARSEABLE_ENDPOINT}
    encoding: proto
    headers:
      X-API-Key: ${env:PARSEABLE_API_KEY}
      X-P-Stream: otel-metrics
      X-P-Log-Source: otel-metrics
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0s
    sending_queue:
      enabled: true
      queue_size: 10000

  otlphttp/parseable_traces:
    endpoint: ${env:PARSEABLE_ENDPOINT}
    encoding: proto
    headers:
      X-API-Key: ${env:PARSEABLE_API_KEY}
      X-P-Stream: otel-traces
      X-P-Log-Source: otel-traces
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0s
    sending_queue:
      enabled: true
      queue_size: 10000

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/parseable_logs]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/parseable_metrics]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/parseable_traces]

Set the environment variables before starting the Collector:

export PARSEABLE_ENDPOINT="https://<YOUR_PARSEABLE_ENDPOINT>"
export PARSEABLE_API_KEY="<YOUR_PARSEABLE_API_KEY>"

otelcol-contrib --config ./config.yaml

PARSEABLE_ENDPOINT is the base Parseable endpoint. The OTLP HTTP exporter appends /v1/logs, /v1/metrics, or /v1/traces for the corresponding pipeline.

If you use Basic Authentication instead of an API key, replace X-API-Key with:

Authorization: "Basic <BASE64_ENCODED_USERNAME_AND_PASSWORD>"

Keep tls.insecure disabled for a production HTTPS endpoint. Configure a custom certificate authority in the exporter TLS settings when your Parseable deployment uses a private CA.

Point applications at the Collector

Applications can send OTLP traffic to either Collector receiver:

For an OTLP HTTP SDK, set:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4318"

Use the Collector Service name that the instrumented workload can reach. Expose the receivers only to clients that need access.

Add other receivers

The OTLP receiver handles telemetry produced by OpenTelemetry SDKs and compatible applications. Infrastructure data often needs additional receivers:

• Use filelog to collect container or host log files.
• Use prometheus to scrape Prometheus metrics endpoints.
• Use hostmetrics for CPU, memory, filesystem, network, and process metrics.
• Use Kubernetes receivers and the k8sattributes processor to add pod, namespace, node, and workload context.

Reference each declared receiver in the appropriate pipeline. A declaration outside service.pipelines leaves the receiver disabled.

For Kubernetes collection, follow the Kubernetes guide.

Verify the pipeline

Check that the Collector process remains healthy:

curl http://127.0.0.1:13133/

Then send telemetry through the Collector and confirm that these datasets receive recent records in Parseable:

You can rename the datasets by changing X-P-Stream, but keep X-P-Log-Source set to the signal-specific value shown in the configuration.

Production guidance

• Put memory_limiter before processors that buffer or transform data.
• Put batch after filtering or sampling processors to avoid batching records that the pipeline will drop.
• Keep exporter retries and sending queues enabled to absorb temporary backend or network failures.
• Size the queue for the expected outage window and available Collector memory. The Collector discards its in-memory queue when it restarts.
• Monitor accepted, refused, sent, failed, and queue-capacity metrics from the Collector's internal telemetry.
• Use multiple Collector replicas for availability. Stateful processors such as tail sampling require additional routing so spans from one trace reach the same Collector.
• Redact credentials and personal data before export. See Remove PII at the Source with OpenTelemetry Collector.
• Load-test the complete pipeline before choosing batch sizes, queue sizes, memory limits, and replica counts for production.

Troubleshooting

The Collector receives no data

• Confirm that the application uses port 4317 for OTLP gRPC or 4318 for OTLP HTTP.
• Confirm that the receiver appears in the corresponding service.pipelines entry.
• Check Services, network policies, firewalls, DNS, and TLS settings between the application and Collector.

Data reaches the Collector but not Parseable

• Check Collector logs for authentication, TLS, timeout, and export errors.
• Confirm that PARSEABLE_ENDPOINT is reachable from the Collector.
• Confirm that the API key has write access to each dataset named by X-P-Stream.
• Confirm that the exporter is enabled in the correct pipeline.

The Collector restarts or drops data

• Check its container memory limit and the memory_limiter thresholds.
• Inspect exporter queue utilization and failed-send metrics.
• Reduce expensive transformations, increase resources, or scale the Collector when export throughput remains below ingestion throughput.

See the OpenTelemetry logs, metrics, and traces documentation for signal-specific details.