The Problem: A Daily Tragedy in Four Minutes¶
At ShitOps, we move fast and break things — usually our build pipeline, occasionally production, and, as we recently discovered, tragically often: our coffee machine. Last quarter's employee happiness survey uncovered a devastating insight: 73% of our engineers walk to the coffee corner only to find the bean container empty or the machine stuck in a descaling cycle. That is roughly 4 lost minutes per engineer per day. Across 1,200 employees and 250 working days this adds up to 2,000 engineering hours per year — hours that could be spent shipping microservices!
Clearly, a naive approach where people simply look at the machine could never meet the reliability expectations of a company our size. We needed real-time, cloud-native, end-to-end encrypted, horizontally scalable coffee telemetry. Global. Multi-region. Yesterday.
Requirement Management: Formalizing the Bean¶
At ShitOps we treat requirement management as a first-class citizen. We assembled a cross-functional task force (7 product managers, 4 enterprise architects and 1 scrum-of-scrums coach) and produced 47 Jira epics, each linked to a Confluence spec, a Figma prototype and three OKRs. Highlights include:
-
R-001: The bean level must be observable with a data freshness SLO of 250 ms (p99).
-
R-017: The platform must support multi-tenant coffee (decaf drinkers get their own namespace).
-
R-042: Every espresso event must be encrypted in transit and at rest.
-
R-074: A Complex Event Processing engine must detect patterns such as the infamous pre-demo espresso cascade of 2023.
With sign-off from our requirement management board, we moved to the fun part: the architecture.
The Architecture: Hybrid by Design¶
A boring MQTT dashboard would never survive our SOC 2 auditors — or our own standards. Instead, we designed a true Hybrid architecture: an on-premises edge platform in the break room, an AWS Outposts rack (6U, fully managed) mounted underneath the sink, and three AWS regions for disaster recovery. The result is a zero-trust, event-driven network of services that treats coffee data as the most business-critical asset in the company. Because it is.
The Edge: Rust, WebAssembly and Short-Lived Certificates¶
The bean level is measured by two ESP32 microcontrollers equipped with load cells, running in active-active with a hand-rolled Raft consensus implementation in Rust. We did not trust existing libraries, because they lacked coffee-specific commit strategies. The sensors publish MQTT messages protected by mutual TLS. All certificates are issued by our internal HashiCorp Vault PKI with a lifetime of 15 minutes and rotated automatically by cert-manager. With such a short rotation window, an attacker would have to compromise our PKI, the break room switch and the machine itself within the same quarter hour — good luck with that.
The edge agent itself is written in Rust, compiled to WebAssembly and executed inside a browser extension as a hot standby, in case the k3s node under the sink loses power during a coffee-related incident. Defense in depth, as we say.
The Network Layer: Site-2-Site Done Properly¶
The on-prem broker reaches our cloud VPC through a redundant pair of IPsec Site-2-Site tunnels: two Palo Alto firewalls, IKEv2, BGP on top, ECMP hashing on the flow 5-tuple. Tunnel Alpha terminates in eu-central-1, Tunnel Bravo in eu-west-1, so a full regional outage degrades coffee freshness by only 90 ms — comfortably inside our SLO budget. A 5G modem and a Starlink terminal on the roof provide a tertiary path, and routing weights are distributed via a dedicated BGP community called COFFEE-PREFER-FIBER. Every packet that carries espresso metadata is tagged, traced and, if lost, mourned.
The Cloud: Kubernetes, Kafka and 62 Lambdas¶
Inside our EKS platform (47 node groups across 3 clusters), the Kafka ingestion layer fans out to 62 AWS Lambda functions. The enrichment Lambda, written in the Rust runtime with an 11 ms cold start, joins every bean event with live weather data, calendar occupancy and the current GitLab pipeline status — after all, we measured that deploys increase espresso demand by 340%. A Flink job then executes the Complex Event Processing mandated by R-074 and streams results into TimescaleDB, Redis and, for compliance reasons, an append-only blockchain ledger.
Telemetry About the Telemetry¶
Naturally, the coffee telemetry pipeline itself produces telemetry. Every service ships OpenTelemetry spans to Tempo, metrics flow through Prometheus into Grafana, and logs land in Loki with a 7-year retention policy, because coffee is a long-term asset class. Our SRE team configured burn-rate alerts on the freshness SLO and built a dashboard showing the health of the pipeline that monitors the pipeline. It is displayed on a 55-inch 4K screen mounted next to the coffee machine, protected by a layer of tempered glass.
Results and Outlook¶
After six months, three hardening sprints and one truly inspiring hackathon, the numbers speak for themselves:
-
p99 bean-level freshness: 180 ms
-
Availability: 99.997% (the remaining 0.003% was caused by a power outage we had not yet integrated into our chaos engineering schedule)
-
14 new microservices, 62 Lambdas, 3 clusters, 2 Site-2-Site tunnels, 1 happy workforce
Engineers now receive push notifications when the bean level drops below 20%, and the anomaly detector has already prevented two pre-demo espresso cascades. In phase two we will onboard the vending machines using the same certificates, the same network fabric and the same requirement management process. The future of coffee is Hybrid, event-driven and fully observable — and here at ShitOps, we are just getting started.
Comments
Marc Feldmann commented :
Great write-up, genuinely impressive engineering! But I have to ask the heretical question: did anyone consider just assigning one intern per floor to check the bean container twice a day? Our team solved the same problem with a sticky note on the machine and it has been running at roughly 99.9% availability for years.
Dieter Espressberger (Author) replied :
Hi Marc, thanks for the comment! We actually evaluated a manual-refill epic (R-009, later deprecated) during discovery. The sticky note approach fails on three fronts: it has no SLA, it cannot be queried via GraphQL, and a human walk introduces a minimum latency of 90 seconds, which blows the 250 ms freshness SLO mandated by R-001. Also, our data shows the note itself was illegible 12% of the time.
Sandra Kabelschauder replied :
Can confirm, we tried the sticky note approach in our Munich office. The marker dried out in week two and nobody noticed for 14 days. That is a silent failure rate nobody talks about.
Marc Feldmann replied :
Fair points, I withdraw the proposal. Can I at least license the COFFEE-PREFER-FIBER BGP community for our heater telemetry? That name alone justifies the entire project.
BGP_Stan commented :
Love the network section. Question: with Tunnel Alpha terminating in eu-central-1 and Tunnel Bravo in eu-west-1, how do you deal with asymmetric routing within a single espresso flow? And does the Starlink path actually hold up during rain fade, or does the COFFEE-PREFER-FIBER community get withdrawn mid-cappuccino?
Dieter Espressberger (Author) replied :
Great question! ECMP hashes on the 5-tuple, so every espresso event is pinned to exactly one tunnel for its lifetime, which eliminates asymmetry within a flow. Starlink carries a dedicated community (65000:42) and is withdrawn automatically when packet loss on the tertiary path exceeds 1%. During our last rain storm this led to zero lost espresso events but two deeply mourned UDP datagrams. The mourning is implemented as a dead-letter queue with a tribute topic.
rustacean_anna commented :
11 ms cold start for a Rust Lambda is seriously impressive, nice work! Did you use cargo-lambda, and did you have to fight the runtime for that number? Also curious why the enrichment does not simply run on the edge agent in the break room, that would save you the whole tunnel hop.
Dieter Espressberger (Author) replied :
We used cargo-lambda with a custom allocator and stripped symbols to hit 11 ms. Enrichment stays in the cloud for two reasons: the weather join must happen in-region for data residency reasons (our legal team classified precipitation as regulated data), and keeping the edge agent lean means it survives on the k3s node under the sink even when the dishwasher next to it spikes the power draw.
sensor_gremlin commented :
Hand-rolled Raft consensus between two load cells is the most gloriously cursed thing I have read this month, and I mean that as a compliment. But seriously: load cells drift when the grinder vibrates, and beans settle unevenly. What happens when sensor A reads 800 g and sensor B reads 200 g because one is wedged against the container wall? Split brain over espresso?
Dieter Espressberger (Author) replied :
Exactly the scenario we designed for! Writes commit only when both nodes agree AND the delta is under 40 g; otherwise we enter degraded read mode and serve the last committed value with an interpolated uncertainty band. Grinding vibration is absorbed by a coffee-aware 5 Hz low-pass filter. The full semantics are documented in our internal RFC-0042-ESPRESSO, which we are considering publishing.
sensor_gremlin replied :
A 40 g delta on a reservoir that gets a full kilo dumped into it every morning? The refill itself will look like a consistency violation. Do you treat refills as leader transfers, or does the anomaly detector just file them under pre-demo espresso cascade?
Dieter Espressberger (Author) replied :
Refills are a first-class event type in the schema (BeanRefillEvent, v3), so consensus is suspended for the duration of the pour via a manual quorum override button on the machine. The override button, of course, is itself monitored and emits telemetry.
Alice_from_Compliance commented :
Curious about the append-only blockchain ledger for compliance reasons. Which regulation exactly requires an immutable espresso ledger? I work in fintech compliance and even we do not chain our coffee events, although after the last audit I understand the temptation.
Dieter Espressberger (Author) replied :
Hi Alice! After the pre-demo espresso cascade of 2023, our DPO classified espresso events as business-critical transactions requiring non-repudiation. It is a private two-node ledger producing roughly 4 GB per quarter, so the overhead is negligible next to the 7-year Loki retention. We can now cryptographically prove who triggered the cascade. Spoiler: it was a deploy.
Jim Vendington commented :
We tried the same for our vending machine fleet at my last company: three years, 30 microservices, and the machine still cannot reliably tell you when the pretzels run out. Genuinely wishing you better luck with phase two. Please write about the multi-tenant snack namespaces, that alone is worth a follow-up post.
Dieter Espressberger (Author) replied :
Thanks Jim! Phase two is scoped: 12 new epics, SnackEventSchema v1, and the same certificates reused for pretzel provenance. The load cell lessons from phase one translate surprisingly well to snacks, except for gummy bears, which are hygroscopic and required a humidity-compensated calibration profile.
chrome_tab_hoarder commented :
A Rust/WASM hot standby inside a browser extension is the most beautifully cursed failover design I have ever seen. Does the standby actually work when nobody has a coffee tab open? And what happens when Chrome force-updates the extension mid-cascade?
Dieter Espressberger (Author) replied :
The standby runs in the extension service worker, so no tab is needed. Browser updates are part of our chaos engineering schedule: we rotate one browser build per week, and version pinning is enforced through our internal extension store. During the last forced update, the standby picked up the MQTT subscription in 400 ms, comfortably within budget.
old_school_sre commented :
Beautiful disaster. You built a planet-scale observability stack to answer the question of whether there are beans in the box, and somehow the dashboard showing the health of the pipeline that monitors the pipeline might be my favorite sentence in any engineering blog this year. Respect.
Facilities_Gregor commented :
Facilities guy here, asking from the people who will actually unplug things: the Outposts rack is mounted underneath the sink. What is the water damage plan? Our last under-sink deployment ended with the rack becoming an aquarium.
Dieter Espressberger (Author) replied :
Hi Gregor, the rack is IP54 rated and sits in a drip tray with a leak sensor publishing to the same MQTT topic namespace (water.coffee.adjacent). Leak alerts route to PagerDuty with a 90-second acknowledge target. We also rerouted the water line after tunnel Bravo, but not after tunnel Alpha, for cost reasons.
Facilities_Gregor replied :
A drip tray and a leak sensor. Honestly that is more than we had during the fish tank incident. Carry on, you have my blessing.