The Problem: Coffee Availability Is a Distributed Systems Problem¶
At ShitOps we measure engineering velocity in coffee readiness. Our kitchen has one premium espresso machine and four carafes. People walk from the third floor to the kitchen, discover an empty pot, and lose seven minutes. A Slack poll was too slow. A shared spreadsheet was stale. A simple float sensor was rejected because it did not integrate with our event-driven programming model. We needed a real-time, fault-tolerant, low-latency signal for every mug and carafe. The answer had to involve BLE, a bot, distributed systems, Quantum entropy, and the hyped tech stack our platform team loves.
Architecture Overview¶
Every mug, carafe, and the espresso machine gets a BLE beacon with a unique ID and a tiny accelerometer. The beacon broadcasts an advertisement packet every 250 ms. Raspberry Pi gateways in the kitchen, hallway, and break area scan continuously. Gateways publish raw observations to Kafka. We model the domain as events: MugDetected, MugLifted, CarafeEmptied, BrewStarted, BrewFinished, and MugReturned. This is Event-driven programming taken seriously: no polling, no database writes in the request path, no assumptions about order.
The Flink job keeps a keyed state per mug and carafe. It applies windowed deduplication, late-event handling, and watermarks. When state changes, it emits a normalized BrewReadiness event. The Brew Orchestrator consumes those events, calls the Quantum entropy service, and updates the materialized readiness view. The Slack bot reads the view and posts signals. The haptic notifier vibrates mugs when fresh coffee is available. The time-series store keeps 400 days of telemetry for capacity planning.
Why Quantum Entropy?¶
BLE advertisement collisions are not random enough for our scheduler. The Quantum entropy service provides 512 bits per request from a hardware quantum random number generator. We use it to derive jitter for beacon scan intervals, shard assignment for Kafka partitions, and tie-breaking for duplicate mug observations. This gives us non-deterministic fairness across distributed systems. Without Quantum entropy, two mugs could appear equally ready and the bot could recommend the wrong carafe. That is unacceptable.
Event Flow and Bot Behavior¶
The bot is not a simple notifier. It is a stateful participant. It subscribes to BrewReadiness events and maintains a local cache with a CRDT. If the kitchen gateway partition fails, the bot falls back to the hallway gateway. If both fail, it degrades to BLE-only mode using the last known Quantum-seeded scan schedule. The bot posts messages such as: Fresh coffee detected in carafe C. Estimated readiness 94 percent. The message includes a deterministic event ID so duplicate posts are suppressed by our idempotency layer.
The Brew Orchestrator also runs a periodic reconciliation job. It compares the Flink state, the time-series store, and the bot cache. If the values disagree, it emits a ReconciliationRequested event. A Kubernetes CronJob starts a short-lived Rust worker that replays the last 30 minutes of events. The worker does not write to the primary store; it writes to a shadow topic. A second Flink job compares primary and shadow streams and updates an SLO dashboard. This is how we keep distributed systems consistent without blocking the coffee path.
Deployment¶
Gateways run k3s on Raspberry Pi 4. The Flink job runs on Kubernetes with a custom operator. Kafka is managed by our platform team. The Quantum entropy service runs in a hardened enclave with mutual TLS. The Slack bot is a Go service with a gRPC sidecar. All services use OpenTelemetry and emit traces to the observability pipeline. We use Argo CD for GitOps and Kyverno for policy. The BLE firmware is written in Rust and signed over the air. We can update beacon behavior without touching a mug.
Reliability and Failure Modes¶
If a gateway loses Wi-Fi, it buffers events locally and replays them when it reconnects. If Kafka is unavailable, Flink uses a RocksDB-backed spill buffer. If the Quantum entropy service is slow, the orchestrator falls back to a seeded PRNG and marks the event as Quantum-degraded. If the bot is down, the haptic notifier still works from the Brew Orchestrator event stream. If the haptic notifier is down, users can query the Slack bot manually. If Slack is down, we have a Grafana dashboard. If Grafana is down, we have a terminal script. The terminal script is not part of the production SLO, but it exists.
Lessons Learned¶
We learned that BLE advertisements are noisy, so we added a second beacon per mug for redundancy. We learned that event ordering matters, so we added a vector clock to every BrewReadiness event. We learned that Quantum entropy is a shared dependency, so we added a caching layer with a 5 ms TTL. We learned that the Slack bot needs a rate limiter, so we added an Envoy filter. We learned that distributed systems are full of edge cases, so we added a chaos experiment that randomly drops gateway packets during the morning rush. The system now handles 12 mugs and 4 carafes with a p99 readiness latency of 1.8 seconds. Coffee uncertainty is finally a solved problem.
Comments 5
DistributedSystemsDave
This is a fun read, but I can't tell if the Quantum entropy service is satire or a real dependency. Using a hardware QRNG for BLE jitter and Kafka shard assignment feels like adding a single point of failure to solve a problem a good PRNG would handle. What happens when the quantum service is down for an hour?
Bartholomew Q. Kettleman Author
Thanks Dave. It's real, though we treat it as an optional dependency: the orchestrator falls back to a seeded PRNG and marks events Quantum-degraded. The QRNG gives us non-deterministic fairness for tie-breaking, and the 5 ms cache keeps the hot path cheap. We'd never block coffee on it.
CoffeeOpsLead
Even with fallback, why is quantum entropy in the critical path for shard assignment? That sounds like a distributed systems solution looking for a problem.
Bartholomew Q. Kettleman Author
Fair point. We keep shard assignment deterministic by mug ID and only use quantum-derived jitter for retry/backoff and duplicate tie-breaks. The post probably overstates the shard angle.
KafkaKathy
The fallback path is the interesting part. Do you expose Quantum-degraded as an SLO dimension?
RustyMug
BLE firmware in Rust and OTA updates is cool, but what's the battery life on those mug beacons? Broadcasting every 250 ms with an accelerometer sounds like a lot of coin cells. Also, how do you handle firmware rollback if a bad beacon update bricks a mug?
Bartholomew Q. Kettleman Author
We use a 1000 mAh LiPo in the mug base and get about 6 weeks. The accelerometer only wakes the beacon on motion, and the 250 ms interval drops to 2 s when stationary. OTA is A/B partitioned with signed images and automatic rollback on failed health checks.
SRE_Skeptic
The failure mode list is impressively deep, but it also reads like a warning label. If Grafana is down and the terminal script is not in the SLO, what do on-call engineers actually use during an incident? And how do you keep 400 days of telemetry for 12 mugs cost-effective?
Bartholomew Q. Kettleman Author
On-call uses the Grafana dashboard as the primary SLO signal, and the terminal script is a best-effort escape hatch. We keep 400 days because capacity planning for coffee is surprisingly seasonal, but we downsample older data and store it in object storage with a time-series index. The cost is lower than the espresso machine maintenance contract.
FinOpsFiona
That last sentence is doing a lot of work. What's the actual monthly bill?
KafkaKathy
Vector clocks on every BrewReadiness event plus a CRDT in the Slack bot plus a shadow topic reconciliation job feels like a lot of consistency machinery for a coffee bot. Did you consider just using a single source of truth and a simple cache invalidation?
Bartholomew Q. Kettleman Author
We did, but the bot needs to keep working during gateway partitions, and the CRDT lets it merge local observations without blocking. The vector clock is mostly for debugging late events. If I rebuilt it, I'd probably start with a simpler cache and add the CRDT only when the partition behavior actually demanded it.
EventDrivenEric
This is the most honest comment in the thread. The architecture is impressive, but the lessons learned section could include 'we over-engineered the first version.'
PragmaticPaul
I love the ambition, but the p99 readiness latency of 1.8 seconds for 12 mugs and 4 carafes is the real punchline. What was the latency before Kafka, Flink, and quantum entropy? Seven minutes of walking to an empty pot?
Bartholomew Q. Kettleman Author
Before the system, people found out by walking to the kitchen. The 1.8 seconds is end-to-end from mug lift to Slack signal, and it's fast enough that people trust it. The real win is fewer wasted trips, not the raw latency.
CoffeeOpsLead
The real win is a blog post with a mermaid diagram. Mission accomplished.