The challenge
Tigris serves 10M+ buckets across 11+ regions, holding 50 PB+ and handling 50B+ requests. Throughput and latency define its service, making per-bucket visibility critical.
The Tigris team adds every new bucket as a label along with other metrics like region and environment. Per-bucket metrics are not optional: they power per-customer SLOs and noisy-neighbor detection.
As the number of buckets grew, the cardinality exploded. Their VictoriaMetrics cluster required multiple TBs of RAM and that requirement kept growing linearly with buckets and cardinality.
Options to scale further meant either significant infrastructure cost & complexity or cutting down on query fidelity and duration. This prompted the team to look for alternative solutions.
Why Parseable
Parseable’s object store native design stores metrics as append-only Parquet tables. In this case, it compresses one hour of standard OTEL metrics to about 250 GB; roughly 50x smaller than raw.
A new series adds a row to existing label columns; only a new label key adds a column. With no in-memory per-series index, higher cardinality has minimal impact on memory and resources.
“We run object storage for a living, so we had opinions about observability built on it. Tigris is a multi tenant system and we want deep visibility into it, so our metrics footprint is huge. Parseable handles that on object storage. We own the data, we set the retention, and scaling is not something we have to worry about. And when we hit issues, the team was on it fast.”
After Parseable
The migration required updating the OTEL Collector configurations to send data to Parseable. All the existing dashboards continued to work without changes, thanks to Parseable’s PromQL support. Alerts continue to run on the existing vmalert rules and routing, now querying Parseable.
Here are the post-migration highlights:
Cardinality independence
- As the number of buckets grows, the Tigris team doesn't see a corresponding increase in memory or resource consumption. This was a primary concern while evaluating Parseable.
- With this, the team is now able to handle their fast growth without worrying about memory or resource constraints.
Ingestion at scale
- The telemetry pipeline produces over 100 million active series. In exact terms, this translates to more than 180M data points per minute, i.e., roughly 300 TB of raw JSON per day.
- To handle this load, currently there are 5 ingestion nodes in the Parseable cluster. They consume a total of 50 vCPU and ~200 GiB, with ~30% headroom.
Faster queries
- Parseable query nodes are stateless, backed by smart, local disk caching. The query plan is cache and object store aware, so it avoids any unnecessary round trips to the object store. Additionally, the Tigris team runs Tigris Acceleration Gateway (TAG) as the write-through cache for better latency.
- This ensures ~25% lower p95 query latency. Bloom filters and Parseable metadata also prune data aggressively. For example, on the top-buckets query, up to 95% of data is skipped. For a top-buckets query across 10M buckets over 1-day period (300 TB raw), the p95 dashboard query latency is ~30 s, versus ~40 s on VictoriaMetrics.
Longer retention with object store
- The team is now able to retain telemetry data for up to a year on the object store. Since the data is heavily compressed, it consumes significantly less storage while remaining queryable.
- This gives them the ability to learn from historical telemetry data and make informed decisions based on long-term trends.
Summary
- Overall RAM no longer grows with bucket count.
- 5 ingestion nodes with 50 vCPU and 200 GiB RAM, ingesting ~300 TB raw JSON per day.
- p95 top-buckets query: ~30 s vs ~40 s on VictoriaMetrics.
- Compute and Storage are cleanly isolated and the architecture allows for elastic scaling up or down as needed.
- Retention increased up to 1 year vs 2 weeks earlier.