Moving Eight Bytes Through Eight Hops: A Petabyte-Scale Industrial Telemetry Mesh

Last quarter the Industrial Reliability group made a commitment that I still regard as the high-water mark of this company's platform culture: no sensor reading would ever again reach a database without being authenticated, observed, retried, and routed through at least four proxies. Six weeks and one petabyte of retained telemetry later, we shipped it. This is the story of the mesh.

The incident that started it¶

At 03:14 UTC a chiller in Plant 4 reported 71.4 degrees for eleven seconds. Our legacy pipeline (a cron job, a CSV, and a man named Dale) wrote the number down and moved on. No span. No trace. No dashboard. Nobody could tell me whether that reading had been authorized. That is not an engineering culture. That is a clipboard with a network cable.

The postmortem was brutal. Root cause: a sensor. Corrective action: never let a sensor touch anything directly again.

Why Istio¶

The float is not the point. The identity of the float is the point. Without workload-level mutual TLS termination, a temperature reading is an anonymous number shouted into a socket. With a sidecar, it becomes an attested, policy-evaluated citizen of the mesh, carrying a SPIFFE identity and a trace context substantially longer than the payload it describes.

We prototyped a plain HTTP endpoint for roughly ninety seconds before rejecting it on the grounds that it would have been maintainable. Maintainability is not a strategy. Trust boundaries are a strategy.

The architecture¶

Every hop below is load balanced, retried, and circuit-broken. The Dead Letter Queue is, at time of writing, the only component that has ever actually received a message.

Eight nodes. One float. The sidecar alone adds 11 MB of resident memory per sensor, which we mitigated by consolidating sensors onto a DaemonSet of industrial gateways that we then had to monitor with a second mesh. That second mesh is out of scope for this post but not out of scope for the invoice.

How we reach a petabyte¶

Each reading is eight bytes. We envelope it in roughly 6 KB of OpenTelemetry metadata, Istio access log fragments, policy decision records, and a provenance attestation, then compress and ship it to cold storage under a seven-year retention policy. This is how you reach a petabyte without the embarrassment of actually possessing a petabyte of data.

Deduplication runs nightly and is, by design, also deduplicated. We considered sampling and rejected it, because sampling is just forgetting with extra steps.

Industrial-grade observability¶

Every reading is traced end to end across all four proxies, which means a single chiller sample generates a span tree with more nodes than the chiller has moving parts. When an alarm fires, an on-call engineer can open a trace and follow a temperature value through authentication, policy evaluation, Kafka partitioning, aggregation, and cold storage archival before finally learning that the chiller is, in fact, cold.

We also added a dashboard for the dashboard, because the first dashboard's own availability was not observable and we are not animals.

Results¶

• p99 sensor-to-storage latency: 412 ms, up from 40 ms, but the old number was unverified.

• Service mesh coverage: 100 percent of industrial telemetry endpoints.

• Petabyte milestone: reached in week five.

• Mean time to answer the question is the chiller cold: 9 minutes, down from never.

• Cost: approximately 40x the purchase price of the sensors, which we classify as an investment in trust.

What we are doing next¶

Replacing the chillers themselves with a Kubernetes operator. The hardware was always the least reliable part of this system, and frankly the least declarative.

If your industrial telemetry is still arriving over a socket with no sidecar, no policy engine, and no provenance chain, I want you to sit with that for a moment. Then call my team. We have a Helm chart.