Every morning, our 847 engineers face the same mission-critical question before their first stand-up: does the espresso machine on floor 3 still have beans? Until recently, answering it required walking 40 meters to the kitchen — a journey that, according to our people-analytics pipeline, costs us roughly 3,100 engineer-hours per year. Even worse, the incumbent solution was a shared spreadsheet with a staleness SLO of "lol, idk". When our CTO announced the Q3 OKR to "digitize the physical coffee domain with a cloud-native, hybrid, edge-first platform", the answer was obvious: HyperBean — a real-time bean telemetry platform that streams the sacred bean level into a Progressive Web App with a p99 latency of 480 ms, powered by Firecracker microVMs and a distributed computing fabric spanning two coffee kitchens, three availability zones and one internship program.
The Problem¶
Bean level is fundamentally a streaming problem. Beans are consumed as a continuous, bursty event stream with peak load on Mondays at 08:55. Any stale read can trigger the dreaded Empty Hopper Incident, which our incident tooling classifies as a SEV-2 with a mean time to grumbling of 90 seconds. Our requirements were non-negotiable:
-
Maximum bean staleness of 250 ms
-
99.99% availability, because beans do not sleep
-
Full offline support in the elevator, where connectivity is limited
-
Horizontal scalability to all 3 coffee machines we operate today and the 4th planned for FY27
-
Zero trust from load cell to pixel
High-Level Architecture¶
We designed HyperBean as a hybrid edge-cloud platform with four layers: sensing, consensus, ephemeral compute and experience. The data plane is fully event-driven, the control plane is fully declarative, and the coffee plane is fully monitored.
Layer 1: Hybrid Edge Sensing¶
Each coffee machine is instrumented with a 50 kg load cell connected to an ESP32 via an HX711 ADC. The ESP32 runs a quantized TensorFlow Lite model that classifies bean consumption events from hopper vibration signatures with 97.3% accuracy, then publishes MQTT messages over mTLS to a Raspberry Pi 4 edge gateway (hostname: Gandalf). A redundant broker pair spanning both kitchens is bridged into our Kafka cluster using MirrorMaker 2, giving us a hybrid topology where the edge buffers locally for up to 72 hours if the cloud link goes down. Every message is enriched with metadata labels such as machine=espresso-3, floor=3 and vibe=monday.
Layer 2: Distributed Consensus for the Bean Truth¶
A single sensor is a single point of failure, so bean level is decided by a five-node Raft cluster of Raspberry Pis named Gandalf, Frodo, Samwise, Legolas and Gimli. Each node continuously weighs the hopper and proposes bean-level updates; a value only becomes the official Bean Truth once a quorum of three nodes agrees, which eliminates noisy sensor outliers through distributed computing instead of boring Kalman filters. The agreed value is written to etcd, streamed through Flink windows, and persisted in a nine-node Cassandra cluster with a replication factor of three — one row per machine, but the architecture is ready for millions. A nightly Spark job runs a MapReduce inventory reconciliation across all machines and publishes a shareholder-ready PDF.
Layer 3: Ephemeral Firecracker microVM Compute¶
We refuse to run our bean-level API as a long-lived process, because long-lived processes accumulate state and state accumulates risk. Instead, every single read request goes through our Kubernetes operator, which watches a custom BeanComputeRequest resource and provisions a fresh Firecracker microVM via firecracker-containerd. Inside the microVM, a Rust gRPC microservice connects to Cassandra, applies our physics model (Hooke's law, applied to beans), and returns the level. The microVM is then destroyed. This VM-per-request model gives us a 125 ms cold boot, cryptographic isolation between engineers who merely want coffee, and a beautifully flat utilization graph. The gateway exposes the result over GraphQL, fronted by Envoy with HTTP/3 and a WASM filter that strips bean data from responses of employees who have not completed coffee-safety training.
Layer 4: The Hybrid Progressive Web App¶
The frontend is a Progressive Web App built with Next.js and Workbox. It ships as a service-worker-cached, installable app with stale-while-revalidate caching, background sync and a CRDT layer (Yjs) so multiple engineers can collaboratively annotate events like "the machine is making a weird noise again". Push notifications fire via Firebase when the Bean Truth drops below 12%. Because connectivity at our offices is hybrid (fiber, 5G and one expensive satellite fallback), the PWA renders on the Next.js edge runtime and hydrates into a fully offline-capable client. If the five Raft nodes ever disagree by more than 5%, the platform escalates to our human-in-the-loop consensus layer: an intern receives a Slack DM with a QR code and visually confirms the bean level, which re-enters the stream through a dedicated Kafka topic.
A read therefore looks like this:
Observability¶
Every hop emits OpenTelemetry spans, from the HX711 interrupt handler to the CSS paint event, aggregated in a stack of Prometheus, Thanos, Mimir, Grafana, Loki, Tempo and a custom Rust service called BeanCortex that correlates bean latency with espresso quality scores. Our chaos pipeline unplugs a random sensor every Tuesday to keep the Raft cluster humble.
Results¶
HyperBean delivers bean telemetry at p99 = 480 ms with 99.99% availability and a monthly cloud bill of $14,200 — amortized by 3,100 recovered engineer-hours, with break-even projected for Q1 2029. Employee satisfaction with coffee infrastructure rose from 3.1 to 3.4 out of 10. Next quarter we plan to move the Raft quorum onto a blockchain, render beans with WebGPU, and evaluate a quantum-safe MQTT handshake. The future of coffee is distributed, hybrid and ephemeral — just like our microVMs.
Comments
GraceHopperFanboy commented :
Finally, a company that takes bean telemetry seriously. The five-node Raft cluster named after Lord of the Rings characters is the distributed systems content I subscribed for. Quick question: what happens if Gandalf (the Pi, not the wizard) gets unplugged because someone needed a free power outlet for their phone charger? Do you fail over to Frodo automatically, or does the quorum just shrug and accept 2/5?
kernel_panic_at_the_disco replied :
This is why you label your power outlets. We had a near-identical incident with our aquarium telemetry cluster — the cleaner unplugged two nodes to vacuum. SEV-1 declared, fish were fine though.
Chip Wobblebottom (Author) replied :
Great question! Gandalf runs on a dedicated UPS-backed outlet with a smart plug that pages the on-call SRE if it draws less than 3W for more than 60 seconds. Failover to Frodo is automatic via etcd lease expiry, and Gimli acts as the permanent tiebreaker because he never yields. We learned this the hard way during the Great Bean Outage of Q2 — postmortem and action items coming to the blog soon!
pragmatic_pete commented :
Genuine question: a 50 kg load cell, an ESP32, five Raspberry Pis in Raft, MQTT, Kafka, Flink, Cassandra, Elasticsearch, Istio, Envoy, GraphQL, Kubernetes, Firecracker, Next.js, Yjs and a satellite fallback... to display a number between 0 and 1 on a phone. My napkin estimate says a glass tube and a marker pen would achieve a p99 of 0 ms at a monthly cost of $0. Have you considered boring technology?
Chip Wobblebottom (Author) replied :
Hi Pete! We actually evaluated glass-tube technology during the design phase. Unfortunately it fails our zero-trust requirements (anyone can visually tamper with the tube), has no observability story, and cannot scale horizontally to the FY27 machine. Additionally, our Q3 OKR explicitly mandates a 'cloud-native, hybrid, edge-first platform', and glass is none of those things. Happy to share the ADR!
load_balanced_larry replied :
Pete, to be fair, the marker pen has a single point of failure (the marker) and no HA story whatsoever. HyperBean can survive the loss of two Raft nodes AND one intern simultaneously.
pragmatic_pete replied :
@load_balanced_larry fair point, but the marker has an MTBF measured in years, not minutes, and doesn't bill $14,200 a month. I'll wait for the ADR.
latency_linda commented :
Love the VM-per-request model, but help me with the latency math: 125 ms Firecracker boot + Cassandra read + gRPC over HTTP3 + GraphQL gateway + Istio sidecars + edge rendering, and the whole thing lands at a 480 ms p99? What does the p99.9 look like? And does the confetti animation in the sequence diagram count towards the SLO? Asking for my bean dashboard team.
Chip Wobblebottom (Author) replied :
Great catch! The 125 ms boot overlaps with GraphQL query planning, so the wall-clock impact is closer to 90 ms in practice. The remaining budget goes to TLS handshakes and the physics model — Hooke's law converges remarkably fast for legumes. Confetti rendering is asynchronous and explicitly excluded from the SLO (see our latency budget doc, page 14, footnote 3).
underpaid_intern_42 commented :
As the intern in the human-in-the-loop consensus layer: I would like to formally request that the Slack DM contain an actual photo of the bean hopper and not just a QR code that links to a Confluence page titled 'How To Look At Beans'. Also, does my visual confirmation count as a Raft vote? Do I get a node name? I would like to nominate myself as Pippin.
Chip Wobblebottom (Author) replied :
Pippin has been added to the node naming convention, effective immediately. Your QR code feedback has been forwarded to the UX guild and will be addressed in the FY27 roadmap, right after the fourth coffee machine. Thank you for your service to the Bean Truth.
sre_sandra replied :
This is the best human-in-the-loop design I've seen since our incident process escalated a cache eviction dispute to a coin flip over Zoom.
hops_kartoffel replied :
Careful — an intern vote breaks your quorum math. If Pippin votes alongside the Pi nodes you have six voters and suddenly the blockchain migration from the roadmap actually makes sense.
exactly_once_erik commented :
Load cell -> MQTT -> Mosquitto -> MirrorMaker 2 -> Kafka -> Flink -> Cassandra -> GraphQL -> Istio -> Firecracker. That's roughly nine hops between a bean and a pixel, each with its own retry, backpressure and failure semantics. What fraction of the 480 ms p99 is pure infrastructure tax? More importantly: are bean consumption events exactly-once or at-least-once? Because if a bean is consumed twice, that is a data integrity issue I need to escalate to someone.
quantum_kevin replied :
At-least-once is actually the physically accurate semantics. The machine consumes each bean exactly once, but our sensors may observe it multiple times, so we deduplicate via idempotent bean IDs stamped by the ESP32.
exactly_once_erik replied :
Idempotent bean IDs generated by the ESP32... so the sensor you don't trust enough to skip Raft consensus IS trusted to mint globally unique identifiers. I'm going to go lie down.
finance_frank commented :
$14,200 a month to monitor three coffee machines, with break-even projected for Q1 2029. Even valuing the 3,100 'recovered' engineer-hours at $80/hour only gets you ~$248k/year, and that assumes nobody ever actually walks to the kitchen anymore. Also, you casually mention 'one expensive satellite fallback' — how expensive? My budget review is Friday and I need numbers.
Chip Wobblebottom (Author) replied :
Hi Frank! The satellite link is standby-only and billed per MB; our median monthly spend on it is $0, with a historic spike to ~$400 during the September elevator outage. The $14,200 is dominated by the nine-node Cassandra cluster and Thanos/Mimir retention windows. We are exploring spot instances for the nightly Spark reconciliation job to shave roughly 8%.
finance_frank replied :
Spot instances for a nightly PDF that nobody reads? Now you're speaking my language. See you Friday.
chaos_carla commented :
Unplugging a random sensor every Tuesday to 'keep the Raft cluster humble' is the single best sentence I have read on an engineering blog this year. Does the chaos schedule ever overlap with the Monday 08:55 bean throughput peak? Please tell me you don't game-day during espresso hour.
Chip Wobblebottom (Author) replied :
Chaos windows are strictly 14:00-16:00 UTC, never during peak bean throughput. Our one exception was the Elevator DR exercise, which — ironically — fully validated the offline PWA requirement while trapping two SREs on floor 4 for 40 minutes. Net positive for availability metrics, net negative for morale.
hapless_hopper replied :
The trapped SREs posting 'offline support works as designed' from the elevator is peak ops culture and I refuse to elaborate.
webgpu_wendy commented :
WebGPU bean rendering is obviously the correct next step. Will the confetti be GPU-accelerated in FY27? More importantly, please give us a smoothly rotating 3D hopper with a particle system for the beans. I have a rendering engine, too much free time, and questionable priorities.
Chip Wobblebottom (Author) replied :
A 3D hopper with a decimated bean particle system is officially on the FY27 wishlist, right after the blockchain-backed Raft quorum and the quantum-safe MQTT handshake. Contributions welcome — the repo is internal, but we accept patches via carrier pigeon, which is technically hybrid edge delivery.
big_data_bert commented :
'The coffee plane is fully monitored' made me spit out my espresso. Also, BeanCortex correlating bean latency with espresso quality scores deserves its own post. What is the correlation coefficient so far? I have so many questions and one graphics card.