The Terrabyte-Scale S3 Telemetry Memory Incident

At ShitOps we collect telemetry from every service, including detailed memory profiles from our serverless fleet. Those profiles land in S3 as immutable objects. Over the last year the archive crossed a terrabyte per day. Product teams wanted to answer a simple question: what is the p99 memory consumption of checkout-api over the last 90 days? Our old path downloaded S3 objects into a single analytics job, loaded them into memory, and printed a table. The job now needs more memory than the node has, and the S3 listing alone feels like a separate incident.

We did what any responsible platform team would do: we built the S3 Telemetry Memory Mesh. It is an event-sourced, stream-native, multi-plane system that treats S3 as the source of truth, telemetry as a first-class product, memory as a queryable dimension, and the terrabyte archive as a routine input. The mesh has one goal: never let a memory question become a memory outage.

flowchart TD subgraph Ingest A[S3 Telemetry Buckets] --> B[SQS Event Queue] B --> C[Lambda Router] end subgraph Stream C --> D[Kafka Telemetry Bus] D --> E[Flink Memory Aggregator] end subgraph Storage E --> F[ClickHouse Query Mesh] E --> G[Iceberg Terrabyte Lake] end F --> H[GraphQL Gateway] G --> H H --> I[Grafana Memory Dashboard] C --> J[Temporal Reconciliation Workflow] J --> K[DynamoDB State Store] K --> E

Ingesting S3 Telemetry Without Touching the Archive

The ingest plane starts with S3 event notifications for every new telemetry object. Events go to SQS, then to a Lambda router. The router validates the object key, attaches a trace ID, and publishes to a Kafka topic partitioned by service and minute. This gives us replayable telemetry and keeps the S3 archive immutable. A Vector sidecar on each producer also forwards local memory counters so we can compare client-side memory with server-side memory.

The Lambda router is deliberately small. It only routes; it does not parse. Parsing happens in Flink, where we can scale independently. We use KEDA to autoscale consumers from the Kafka lag, and Envoy sidecars to apply mTLS and retry budgets. The router writes a receipt to DynamoDB so Temporal can reconcile every S3 event exactly once. If an event is missing, Temporal starts a backfill workflow. If an event is duplicated, the receipt makes it harmless. This is how we keep a terrabyte-scale telemetry stream trustworthy.

Flink is the heart of the mesh. It consumes telemetry records, extracts memory samples, and computes rolling p50, p95, and p99 memory by service, version, region, and deployment ring. The aggregator writes two outputs: a low-latency ClickHouse table for dashboards and an Apache Iceberg table in S3 for long-term analysis. Iceberg gives us schema evolution and time travel, so a memory regression can be compared against any previous snapshot. We also persist a compact memory sketch in Redis for the GraphQL gateway, because sub-second p99 memory lookups are now a product requirement.

The Flink job uses event time and watermarks. Late telemetry is routed to a side output and replayed through a Temporal workflow. Checkpoints go to S3, and savepoints are tagged with the release train. That means we can upgrade the aggregator during business hours without losing a single memory sample. The terrabyte archive is no longer a query bottleneck; it is a replayable input to a streaming system.

Querying Terrabytes with ClickHouse and GraphQL

The query plane is a GraphQL gateway backed by ClickHouse and Iceberg. Dashboards ask for memory percentiles and receive a typed response. The gateway resolves recent windows from ClickHouse and historical windows from Iceberg through a Trino federation layer. We cache query plans in Redis and enforce per-tenant rate limits with Envoy. Every query emits OpenTelemetry spans, so we can see the telemetry query itself as telemetry. This recursive observability is exactly what a mature platform needs.

For expensive terrabyte scans, the gateway creates a Temporal workflow instead of blocking the request. The workflow fans out partition reads, writes a materialized result to S3, and notifies the UI through a WebSocket. The user gets a progress indicator and a signed result object. This keeps memory usage predictable on the gateway and pushes heavy work into the mesh. We call this async query materialization, and it has become the default for any window larger than seven days.

Memory Safety and Operational Guardrails

Every component has a memory budget. The Lambda router is capped at 512 MB, Flink task managers use 8 GB heaps with off-heap RocksDB, ClickHouse nodes use cgroup limits, and the GraphQL gateway runs with a 1 GB limit. Prometheus scrapes memory metrics from every pod, and Grafana alerts when heap usage crosses 70 percent for five minutes. PagerDuty routes those alerts to the Telemetry Memory Guild. We also run a chaos experiment that replays a synthetic terrabyte telemetry day through the mesh every Friday.

Temporal owns the state machine for ingestion, backfills, and async queries. DynamoDB stores idempotency keys and workflow pointers. If any stage fails, Temporal retries with exponential backoff and a dead-letter queue. The mesh is self-healing within the boundaries we defined, and every failure is visible in the same telemetry system it protects. That is the kind of closed loop I want in a platform.

Rollout and Results

We rolled the mesh out behind a feature flag, starting with checkout-api and then moving all services. The first week processed 1.2 terrabytes of telemetry and served 40,000 memory queries. The p99 memory dashboard now loads in 900 ms, and the old download-and-aggregate job is gone. The S3 archive remains the source of truth, while Flink, ClickHouse, Iceberg, Temporal, and GraphQL provide the query surface. We can add a new memory dimension without rewriting the pipeline, and we can replay any terrabyte day to validate a regression.

This is the architecture I would choose again. It respects S3, treats telemetry as a product, makes memory observable, and scales to the terrabyte era. If you are still loading S3 telemetry into a single process to answer memory questions, you are one incident away from a very educational afternoon.