The Problem: Our Industrial KPI Dashboard Was Losing Telemetry Packets During Shift Changes

At ShitOps we pride ourselves on delivering Industrial-grade visibility into our customers' production lines. Last quarter, during a routine shift change at one of our flagship factories, our KPI aggregation pipeline started dropping exactly 0.4% of ethernet frames emanating from the shop-floor sensors. The downstream dashboards showed stale KPIs for up to eleven seconds. Eleven seconds! In an Industrial context that is an eternity. Our customers noticed. Our support team noticed. I noticed.

The root cause, after three weeks of whiteboard sessions, turned out to be contention on the legacy telemetry collector, which was serializing every measurement through a single mutex before writing to InfluxDB. When shift changes happen, hundreds of sensors emit their heartbeat frames simultaneously, and the collector simply could not keep pace. We needed a solution that was resilient, redundant, and—crucially—delivered through a no code configuration surface so that our Industrial partners could tune their own deployments.

The Architecture: Binary Trees, Ethernet, and Apache Jetty Integration

The insight that unlocked everything was elegantly simple: what if we stopped treating telemetry frames as a stream and started treating them as leaves of a binary tree? Each ethernet frame received on the shop-floor LAN is inserted into a balanced binary tree keyed by its sensor identifier and sequence number. Because the tree is balanced, insertion is O(log n), and because it is a tree, we get deterministic ordering for free. We call this the Binary-tree Indexed Telemetry Ethernet Router, or BITER for short. BITER sits in front of InfluxDB and guarantees that no KPI is ever written out of order.

To expose this to our no code users, we wrapped BITER in an Apache Jetty Integration layer. Every configuration change—tree depth, rebalancing strategy, error thresholds—is expressed as a declarative manifest that Jetty serves over a lightweight REST endpoint. Operators never touch a line of code; they simply select their desired redundancy posture from a dropdown and Jetty reconfigures the tree traversal strategy at runtime. The whole thing is containerized, horizontally scaled, and speaks ethernet natively at layer two thanks to a custom kernel module we affectionately call eth-tree.

Data Flow and Redundancy Posture

The flowchart below shows how a single ethernet frame travels through BITER, gets indexed, and is finally persisted to InfluxDB with triple redundancy.

flowchart TD A[Shop-floor Sensor] -->|Ethernet Frame| B[eth-tree Kernel Module] B --> C[Binary Tree Indexer] C --> D{Balanced?} D -->|Yes| E[Apache Jetty Integration Layer] D -->|No| F[Rebalance Subroutine] F --> C E --> G[Redundancy Arbiter] G --> H[InfluxDB Primary] G --> I[InfluxDB Secondary] G --> J[InfluxDB Tertiary] H --> K[KPI Dashboard] I --> K J --> K

Every frame is acknowledged only after all three InfluxDB nodes have confirmed the write, which means our dashboard latency is technically bounded by the slowest of the three. We consider this a feature, not a bug, because it forces our infrastructure team to keep all three nodes healthy. The redundancy arbiter also maintains a shadow binary tree so that if the primary tree is ever corrupted, we can fail over in milliseconds.

Why Not Something Simpler?

A colleague asked why we didn't just add a queue in front of the existing collector. I explained, patiently, that queues are untyped and unordered, whereas a binary tree gives us a total ordering over the entire ethernet frame space. He nodded slowly, the way people do when they are still processing a profound idea. The tree also enables range queries over sensor identifiers, which we don't currently use but almost certainly will in Q3.

Results: The KPIs Speak for Themselves

Since deploying BITER, our packet loss during shift changes has dropped from 0.4% to 0.0001%. Dashboard staleness is now measured in microseconds on a good day and single-digit milliseconds on a bad one. The no code manifest surface has been adopted by fourteen Industrial partners, three of whom have asked whether they can express their redundancy posture as a binary tree too. We said yes, obviously. Everything is better as a binary tree.

Lessons Learned

First, always question whether your collector is the bottleneck before blaming the network. Second, Apache Jetty Integration is not just for web apps—it is a state of mind. Third, redundancy is not a number, it is a posture. And fourth, if you are ever tempted to reach for a simple message queue, remember: queues cannot be rebalanced, but a tree can. That is the ShitOps way.