Storage

Parseable is a purpose built telemetry datalake. This means telemetry data is persisted in inexpensive, infinitely scalable commodity storage such as Amazon S3, Google Cloud Storage, Azure Blob, or other S3‑compatible service (Tigris, MinIO, Wasabi, DigitalOcean Spaces, etc.).

Architecture

Parseable uses Apache Arrow and Parquet as its underlying data structures, optimized for analytical workloads. This columnar format provides:

• Compression efficiency: Significantly reduced storage costs
• Query performance: Fast analytical queries over compressed data
• Schema evolution: Flexible data structure changes over time
• Cross-platform compatibility: Standard format readable by many tools

(SSE)

• Customer-Managed Keys: SSE-C for custom encryption keys
• TLS in Transit: Secure data transmission
• Access Control: Fine-grained permissions through cloud IAM

Supported providers

Parseable supports multiple cloud storage providers and S3-compatible services:

Cloud Providers

• AWS S3: Native integration with all AWS regions
• Azure Blob Storage: Full support for Azure storage accounts
• Google Cloud Storage: Compatible through S3 API

S3-Compatible Services

• MinIO: Self-hosted object storage
• Wasabi: Cost-optimized cloud storage
• DigitalOcean Spaces: Developer-friendly object storage
• Backblaze B2: Affordable cloud storage

Authentication Models

Parseable supports multiple authentication mechanisms to fit different deployment scenarios:

Static Credentials

• Access keys and secret keys for direct authentication
• Suitable for development and simple deployments
• Requires careful credential management

Dynamic Credentials

• IAM Roles: For AWS EC2/ECS deployments
• Instance Metadata Service (IMDS): Automatic credential rotation
• Container Credentials: For containerized environments
• Azure AD Integration: Service principal authentication

Security Features

• Encryption at Rest: Support for server-side encryption (SSE)
• Customer-Managed Keys: SSE-C for custom encryption keys
• TLS in Transit: Secure data transmission
• Access Control: Fine-grained permissions through cloud IAM

Data Organization

Parseable organizes data in object storage using a hierarchical structure:

bucket/
├── streams/
│   ├── app-logs/
│   │   ├── year=2024/
│   │   │   ├── month=01/
│   │   │   │   ├── day=15/
│   │   │   │   │   └── data.parquet
│   └── system-logs/
└── metadata/
    └── schemas/