Every first Friday of the month, something magical happens at ShitOps: Burger Friday. More than 4,200 employees across our 14 global offices fire up their iPhones and place burger orders for our legendary office grill sessions. What most colleagues never see is the amount of infrastructure required to move a perfectly grilled patty from the grill to the mouth without a single point of failure. Today I want to share how we rebuilt this business-critical process from the ground up with AI, version control, and a fully geo-redundant business continuity plan (BCP).
The Problem: A Single Point of Failure Between Us and Our Burgers¶
For years, burger ordering at ShitOps ran on a shared Excel spreadsheet on a network drive. It worked — until it did not. Last quarter the file hit 65,000 rows and corrupted right in the middle of order peak, taking 312 orders (and two office friendships) with it. Our post-incident review uncovered uncomfortable truths:
-
No version control: Nobody could tell us who had changed extra pickles to no pickles in the secret sauce document.
-
No AI: Grill scheduling was based purely on the gut feeling of Dave from facility management.
-
No BCP: Our recovery time objective for burger ordering was measured at 6 hours. Our recovery point objective was we are not sure.
-
No mobile-first strategy: Employees filled out orders on their iPhones, then read them aloud to a colleague who retyped them into the sheet.
We realized that burger ordering is not a nice-to-have. It is a tier-0 business process that deserves the same engineering rigor as our payments platform.
Our Vision: BurgerFlow¶
We assembled a cross-functional tiger team of 9 engineers, 2 data scientists, 1 head chef, and Dave. After three design sprints and one offsite (catered, unsurprisingly, with burgers), we agreed on five non-negotiable requirements:
-
Zero-downtime, iPhone-first ordering from all 14 offices
-
AI-driven demand forecasting with per-office, per-15-minute granularity
-
Burgers as Code: every recipe under strict version control with GitOps deployment
-
A business continuity plan (BCP) with multi-region failover in under 60 seconds
-
Immutable and auditable patty provenance
The result is BurgerFlow, our cloud-native burger ordering platform.
Architecture Overview¶
The diagram below shows the happy path of an order and how our recipe pipeline and regional failover fit together.
iPhone-First Edge Ordering¶
Every employee iPhone is now a first-class edge node in our architecture. The native Swift app BurgerFlow Edge cryptographically signs every order with Face ID — and we did not stop at biometric unlocking: we built a custom signing scheme on top of three stacked HMAC layers for defense in depth. Orders travel via gRPC over QUIC to our Envoy gateway, so a bacon double cheeseburger reaches the grill scheduler in under 90 milliseconds.
If connectivity drops, the app queues orders in a local CRDT that merges conflict-free, even when two colleagues at the same table order the identical burger simultaneously. The App Store review took four months, but the payoff is worth it. Android users are fully supported through a platform-agnostic React Native web view that renders the exact same UI, because inclusion matters to us.
AI-Powered Demand Forecasting¶
At the heart of BurgerFlow sits our Demand Forecast AI, a fine-tuned 67-billion-parameter language model trained on five years of anonymized order history, local weather data, and — this is the real secret sauce — Slack message sentiment. It turns out that the emoji sequence 😩🔥 in the #general channel correlates with a 23 percent bacon demand spike.
The model emits a demand vector for every office in 15-minute windows. Our Grill Optimizer then runs a genetic algorithm to schedule patty batches and minimize what we internally call patty-to-mouth latency. Dave reviews and approves every optimization suggestion, because we firmly believe in humans in the loop.
Burgers as Code: Version Control for the Secret Sauce¶
The secret sauce recipe used to live in a Word document on a laptop that no longer boots. Today, every burger is a semantic version. Our flagship Classic ShitBurger is proudly at v4.2.0. Recipes are declarative YAML manifests in a dedicated Git repository, validated by JSON Schema, peer reviewed via pull requests, and protected by CODEOWNERS — the head chef must approve every patty change.
Deployment follows strict GitOps: ArgoCD syncs recipe releases to the Grill Controller, which applies them to the physical grills. New recipes ship as canary releases covering 5 percent of patties while we monitor patty telemetry such as weight, grill-mark density, and bite resistance. If a release misbehaves, a single git revert rolls back lunch in eleven seconds. Everything is recorded in an append-only audit log, because lettuce traceability must be immutable.
A Business Continuity Plan You Can Taste¶
Burgers are too important to lose, so BurgerFlow runs on three Kubernetes clusters on three different cloud providers across three continents. Kafka MirrorMaker continuously replicates order streams between the regions, giving us a measured recovery time of 42 seconds — verified in monthly GameDays where we simulate full region outages.
Because we practice defense in depth, we also maintain an analog fallback path: a laminated order form and a factory-reset iPhone 4 stored in a glass box in the HQ lobby. We test this break-glass procedure quarterly, and it once saved a grill session when a raccoon tripped the main breaker. On Tuesdays, our Chaos Monkey randomly deletes grill jobs to prove that the platform self-heals.
Results¶
| Metric | Before | After |
|---|---|---|
| Burger availability | 91.2 % | 99.995 % |
| p99 patty-to-mouth latency | 47 min | 12 min |
| Recipe rollback time | ask Dave | git revert (11 s) |
| Microservices | 0 | 14 |
The numbers speak for themselves: 14 microservices, 6 ML models, 23 Git repositories, and one very happy workforce. Dave was promoted to Burger Reliability Engineer.
What Is Next¶
Phase 2 is already in flight: generative AI burger recipes reviewed by a second AI, drone-based cross-office patty transfer, and a WebAssembly runtime in the patty press firmware. We are hiring across all burger-related teams — bring your iPhone.
Comments
Marcus Webb commented :
Am I the only one who thinks a 67-billion-parameter language model is slightly overkill for predicting how many burgers 200 people in the Munich office will eat between 12:00 and 12:15? A Poisson distribution and Dave's gut feeling got you to 91.2% availability already.
Chad Stackhausen (Author) replied :
Hi Marcus, thanks for the comment! You are thinking in single-model terms. The 67B model does not just forecast demand, it contextualizes Slack sentiment, weather, sprint review calendars, and the emotional aftermath of quarterly all-hands meetings. We measured a 34% improvement in bacon spike prediction over a plain Poisson baseline. Also, Dave's gut feeling is now an input feature, encoded as a one-hot vector.
Sarah Okafor replied :
'Dave's gut feeling as a one-hot vector' is the funniest thing I have read in an engineering blog this year. Please tell me it is in the append-only audit log.
Priya Raghavan commented :
Great write-up! Quick question on the BCP: you replicate Kafka across three continents via MirrorMaker, but what happens to the grill itself during a regional failover? The physical grill in Berlin does not care that US-East now owns the order stream. How do you prevent split-brain patties?
Chad Stackhausen (Author) replied :
Excellent question, Priya! Grill state is synchronized through the Grill Controller, which holds an exclusive lease on the physical grill hardware. Only the region holding the grill lease can dispatch patty jobs, so failover is safe by construction. We call this the Patty Consensus Protocol. A deep-dive post on it is coming in Q3.
devops_dan replied :
Patty Consensus Protocol. This entire post is a masterpiece. Honestly, the laminated form plus a factory-reset iPhone 4 in a glass box sounds more reliable than half the systems I have operated in my career.
Priya Raghavan replied :
Thanks for the reply! So it is basically Raft, but for burgers. Looking forward to the Q3 post, I want to see the mermaid diagram for split-brain patties.
Hannah Kim commented :
'git revert rolls back lunch in eleven seconds' is the greatest sentence ever written on an engineering blog. My question: what does a MAJOR version bump of a burger actually mean? If the Classic ShitBurger went from v3.x to v4.2.0, did someone remove the cheese in a breaking change?
Chad Stackhausen (Author) replied :
Great catch, Hannah! v4.0.0 was indeed a breaking change: we renamed 'burger sauce' to 'secret sauce' in the YAML schema, which changed the field name and therefore the API contract with the Grill Controller. v4.1.0 introduced pickles as an optional field, and v4.2.0 tuned the grill-mark density defaults. The full changelog lives in the recipe repo, and every release tag is signed by the head chef with his personal GPG key.
Tomasz Nowak replied :
So a breaking change in the burger API was a schema field rename. I have worked at places where that kind of thing shipped to production on a Friday at 5pm with zero changelog and no GPG key in sight. Respect.
Alex Chen commented :
Security reviewer hat on: three stacked HMAC layers on top of Face ID-signed orders? What threat model exactly are you defending against? Who at your company is forging burger orders, and more importantly, why? Also, is the secret sauce recipe repo public now? Asking for a friend.
Chad Stackhausen (Author) replied :
Hi Alex! Defense in depth. In 2022 an intern emoji-reacted a colleague's order and a double bacon was silently upgraded to a triple without consent. Since then we treat order integrity as tier-0. The secret sauce repo is private, gated by CODEOWNERS, and the head chef must approve any diff touching sauce.yaml. We also run an annual red-team exercise where an internal team attempts to smuggle pineapple onto a burger. Nobody has succeeded yet.
Alex Chen replied :
A red team whose objective is to smuggle pineapple onto a burger is the only red team exercise I have ever genuinely wanted to join. Please publish the write-up.
Dave (Burger Reliability Engineer) commented :
Dave here. Yes, the gut feeling scheduling worked for nine years, and yes, I was skeptical of the AI. But I will admit the Grill Optimizer caught a lunch rush after an all-hands that I would have missed. Also, the raccoon incident was NOT my fault, the breaker cabinet door was already broken. Proud to be a Burger Reliability Engineer now. I have a lanyard.
Kyle from Ops replied :
Dave with the lanyard. The real tier-0 infrastructure was the spreadsheets we corrupted along the way. Congrats on the promotion, king.
Chad Stackhausen (Author) replied :
Dave is being modest. His gut feeling is feature #1 in the demand vector and he reviews 100% of Grill Optimizer suggestions. Humans in the loop is not a buzzword here, it is Dave.
SRE Sarah replied :
Dave, as a fellow on-call engineer: how does the pager rotation work for grill incidents? Is there a Burger Ops channel pinging at 3am, or is that what the three-continent failover is for?
Andy (Android Developer) commented :
'Android users are fully supported through a platform-agnostic React Native web view that renders the exact same UI, because inclusion matters to us.' As an Android developer, I am crying and laughing at the same time. A web view is not support, it is a hostage situation.
Chad Stackhausen (Author) replied :
Hi Andy! The web view is a fully supported first-class client. It even reuses the same gRPC-over-QUIC bridge through a thin Swift shim we compile for Android. A fully native Android app is on the roadmap for Phase 3, right after the WebAssembly patty press firmware. Inclusion matters, and so does roadmap sequencing.
FlutterFanGreg replied :
'A thin Swift shim we compile for Android.' I need to lie down. This post is a flawless simulation of every platform decision meeting I have ever sat in.
Andy (Android Developer) replied :
Chad, I honestly cannot tell anymore whether this post is satire or my last performance review.
Tomasz Nowak commented :
Serious question: a p99 patty-to-mouth latency of 47 minutes before, was that not just people standing in line at the grill? Did you really need 14 microservices and a three-continent Kafka deployment to fix a queueing problem? A second grill would have cost a fraction of the budget.
Chad Stackhausen (Author) replied :
Hi Tomasz! We actually evaluated a second grill under the internal name Horizontal Patty Scaling. Unfortunately grill throughput is constrained by the physical flip rate of Dave, and vertical scaling of Dave was not an option. So we scaled horizontally in software. Also, the old Excel file lived on a network drive that blue-screened every second Tuesday, so the queueing was honestly the least of our problems.
SRE Sarah replied :
'Vertical scaling of Dave was not an option' is going straight into my next capacity planning document, verbatim, no notes.
OldSchoolOlive commented :
I have been running lunch logistics for a 300-person office with a shared spreadsheet and a whiteboard for a decade. Availability: 100%. Uptime: every single Friday. Cost: zero. What exactly did 4,200 employees and 14 offices break that required Kubernetes on three continents? Sincerely, someone whose lunch has never once paged her at 3am.
Chad Stackhausen (Author) replied :
Hi Olive! Respectfully: your solution has no audit trail, no recipe provenance, and no credible path to 99.995% availability. When your whiteboard gets erased, your RPO is 'everything since last Friday.' We prefer measurable resilience. That said, the laminated form in our glass box is essentially your whiteboard, so we have met in the middle. Ours is just versioned, geo-replicated, and tested quarterly.
devops_dan replied :
'When your whiteboard gets erased, your RPO is everything since last Friday' is going directly into my incident review templates. Thank you for your service, Chad.
Keiko Tanaka commented :
Love the Chaos Monkey deleting grill jobs on Tuesdays. One question: has it ever misfired on an actual Burger Friday? Our chaos tooling once picked the exact minute of a product launch, and I would like to believe burger infrastructure has better guardrails than we had.
Chad Stackhausen (Author) replied :
Hi Keiko! Great question. Chaos Monkey is excluded from Burger Fridays via a feature flag, which is itself version-controlled and CODEOWNERS-protected by Dave. The one time it misfired on a Friday, the analog fallback path (laminated form plus iPhone 4) saved the grill session and we wrote a full retro. The raccoon was a separate incident and is not covered by the flag.
GrillDad replied :
The raccoon deserves its own postmortem. That animal achieved in four seconds what no red team member has managed in three annual exercises.
Yusuf Demir commented :
Four months in App Store review for a burger app. FOUR MONTHS. What did you possibly tell the review team when they asked why ordering lunch requires Face ID cryptography and a 42-second regional failover?
Chad Stackhausen (Author) replied :
Hi Yusuf! We submitted 14 demo videos showing the full order lifecycle, including a scripted region-failover demo. The reviewer eventually asked us to remove the phrase 'business-critical' from the App Store description, and we compromised on 'business-important.' The CRDT offline queue also triggered a follow-up review, because the reviewer placed a burger through the demo environment and then wanted to know when it would arrive.
Marcus Webb replied :
The App Store reviewer ordering a burger through the demo and asking when it arrives is the single most realistic App Store story I have ever heard. Everything about this post checks out.