The Problem: A Daily Tragedy in Four Minutes

At ShitOps, we move fast and break things — usually our build pipeline, occasionally production, and, as we recently discovered, tragically often: our coffee machine. Last quarter's employee happiness survey uncovered a devastating insight: 73% of our engineers walk to the coffee corner only to find the bean container empty or the machine stuck in a descaling cycle. That is roughly 4 lost minutes per engineer per day. Across 1,200 employees and 250 working days this adds up to 2,000 engineering hours per year — hours that could be spent shipping microservices!

Clearly, a naive approach where people simply look at the machine could never meet the reliability expectations of a company our size. We needed real-time, cloud-native, end-to-end encrypted, horizontally scalable coffee telemetry. Global. Multi-region. Yesterday.

Requirement Management: Formalizing the Bean

At ShitOps we treat requirement management as a first-class citizen. We assembled a cross-functional task force (7 product managers, 4 enterprise architects and 1 scrum-of-scrums coach) and produced 47 Jira epics, each linked to a Confluence spec, a Figma prototype and three OKRs. Highlights include:

With sign-off from our requirement management board, we moved to the fun part: the architecture.

The Architecture: Hybrid by Design

A boring MQTT dashboard would never survive our SOC 2 auditors — or our own standards. Instead, we designed a true Hybrid architecture: an on-premises edge platform in the break room, an AWS Outposts rack (6U, fully managed) mounted underneath the sink, and three AWS regions for disaster recovery. The result is a zero-trust, event-driven network of services that treats coffee data as the most business-critical asset in the company. Because it is.

flowchart TD S1[ESP32 Load-Cell Sensor A] -->|MQTT over mTLS| B[On-Prem Kafka Broker] S2[ESP32 Load-Cell Sensor B] -->|MQTT over mTLS| B B --> E[Rust Edge Agent on k3s] E -->|IPsec Site-2-Site Tunnel Alpha| GW[AWS Transit Gateway] E -->|IPsec Site-2-Site Tunnel Bravo| GW E -.->|5G Failover Path| GW GW --> K[EKS Kafka Cluster] K --> L1[Lambda Enrichment in Rust] K --> L2[Lambda Anomaly Detector] L1 --> TS[(TimescaleDB)] L2 --> PD[PagerDuty] TS --> GQL[GraphQL Federation Gateway] GQL --> MOB[Flutter Coffee App] GQL --> SB[Slack Bean Bot]

The Edge: Rust, WebAssembly and Short-Lived Certificates

The bean level is measured by two ESP32 microcontrollers equipped with load cells, running in active-active with a hand-rolled Raft consensus implementation in Rust. We did not trust existing libraries, because they lacked coffee-specific commit strategies. The sensors publish MQTT messages protected by mutual TLS. All certificates are issued by our internal HashiCorp Vault PKI with a lifetime of 15 minutes and rotated automatically by cert-manager. With such a short rotation window, an attacker would have to compromise our PKI, the break room switch and the machine itself within the same quarter hour — good luck with that.

The edge agent itself is written in Rust, compiled to WebAssembly and executed inside a browser extension as a hot standby, in case the k3s node under the sink loses power during a coffee-related incident. Defense in depth, as we say.

The Network Layer: Site-2-Site Done Properly

The on-prem broker reaches our cloud VPC through a redundant pair of IPsec Site-2-Site tunnels: two Palo Alto firewalls, IKEv2, BGP on top, ECMP hashing on the flow 5-tuple. Tunnel Alpha terminates in eu-central-1, Tunnel Bravo in eu-west-1, so a full regional outage degrades coffee freshness by only 90 ms — comfortably inside our SLO budget. A 5G modem and a Starlink terminal on the roof provide a tertiary path, and routing weights are distributed via a dedicated BGP community called COFFEE-PREFER-FIBER. Every packet that carries espresso metadata is tagged, traced and, if lost, mourned.

The Cloud: Kubernetes, Kafka and 62 Lambdas

Inside our EKS platform (47 node groups across 3 clusters), the Kafka ingestion layer fans out to 62 AWS Lambda functions. The enrichment Lambda, written in the Rust runtime with an 11 ms cold start, joins every bean event with live weather data, calendar occupancy and the current GitLab pipeline status — after all, we measured that deploys increase espresso demand by 340%. A Flink job then executes the Complex Event Processing mandated by R-074 and streams results into TimescaleDB, Redis and, for compliance reasons, an append-only blockchain ledger.

Telemetry About the Telemetry

Naturally, the coffee telemetry pipeline itself produces telemetry. Every service ships OpenTelemetry spans to Tempo, metrics flow through Prometheus into Grafana, and logs land in Loki with a 7-year retention policy, because coffee is a long-term asset class. Our SRE team configured burn-rate alerts on the freshness SLO and built a dashboard showing the health of the pipeline that monitors the pipeline. It is displayed on a 55-inch 4K screen mounted next to the coffee machine, protected by a layer of tempered glass.

Results and Outlook

After six months, three hardening sprints and one truly inspiring hackathon, the numbers speak for themselves:

Engineers now receive push notifications when the bean level drops below 20%, and the anomaly detector has already prevented two pre-demo espresso cascades. In phase two we will onboard the vending machines using the same certificates, the same network fabric and the same requirement management process. The future of coffee is Hybrid, event-driven and fully observable — and here at ShitOps, we are just getting started.