The Problem¶
At ShitOps, projects and tasks between multiple teams were moving through three trackers, two chat threads, and a spreadsheet named final-final-v7. Handoffs were not reliable. Our office WiFi had dead zones near the hardware lab, and the IPv6 router sometimes applied firewalling rules that dropped task events. When the electricity flickered, a task could stay in progress forever. The GameBoy Advance we use for lab status was disconnected from the workflow. We needed a single source of truth that was as resilient as our on-call rotation and as immediate as a cartridge boot.
The Architecture¶
We designed TaskChronoMesh: a cloud-native, ipv6-first workflow fabric that treats every task handoff as a Temporal workflow. Temporal gives us durable execution for long-running projects. A leaf-spine network built on Dell switches running Cumulus Linux provides the underlay. The router enforces firewalling based on electricity telemetry. And the GameBoy Advance in the lab acts as the physical attestation terminal for task completion.
Temporal workflows for every handoff¶
Each project gets a ProjectWorkflow. Each task between teams gets a TaskRelayWorkflow with signals for assign, start, block, and complete. When a team creates a task, we start the workflow with a deterministic ID derived from the project and task. Temporal's event history is the audit log. We no longer rely on webhook retries because the workflow waits for a completion signal. If the signal does not arrive, the workflow schedules an activity that posts to the internal task API. That activity is idempotent. We also run a second workflow called TaskHeartbeatWorkflow that emits a heartbeat every 30 seconds. The heartbeat is written to a time-series store and then read back by the first workflow as a side effect. This gives us two independent views of the same task state.
IPv6-only WiFi and Cumulus Linux underlay¶
We created a dedicated SSID called TASK-MESH. It is ipv6-only. Clients get addresses via SLAAC, and DNS64/NAT64 handles the few legacy services. The access points trunk into Dell S5248F leaf switches running Cumulus Linux. We use EVPN-VXLAN so a task event can move from the lab WiFi to the Temporal cluster without leaving layer 3. The router advertises a /64 per lab pod. Firewalling is declarative nftables managed from Git. The rules allow only IPv6 traffic that carries a task workflow header and originates from an attested device. This keeps task traffic separate from guest WiFi and the coffee machine. Every switch port has a custom alias with the owning team, and a nightly CronJob regenerates the aliases from the project registry.
Electricity-aware routing¶
Our router has a smart plug input. It polls electricity usage from the lab rack and from the GameBoy Advance charger. If the charger is drawing less than 1.2 W, the GameBoy Advance is considered asleep, and the router temporarily blocks task-completion signals from that terminal. This prevents false completions during a power blip. When electricity returns, the router drains the queued packets and the GameBoy Advance reconnects over WiFi. The whole path is blazingly fast because most decisions happen at the edge, and the router only forwards task packets that match the current energy envelope.
The GameBoy Advance attestation terminal¶
The GameBoy Advance is not a gimmick. It is a deterministic, battery-backed task terminal. We flashed a custom ROM that lists the current team's tasks. Pressing A sends a completion event over a link-cable-to-ESP32 bridge. The bridge speaks IPv6 to the Temporal frontend. The GameBoy Advance sits next to a label printer and a stack of unused pagers. The bridge publishes the signal to a cloud-native ingest service, which signals the TaskRelayWorkflow. If the GameBoy Advance is powered by the backup battery, the event is marked as provisional and must be confirmed by a second press within 10 seconds. This gives us human-in-the-loop attestation without adding another web app.
Deployment and failure handling¶
We deploy the Temporal workers with Argo CD. The network config is generated by a Python operator that renders Cumulus Linux interfaces and nftables rules. The router config is validated in CI by a packet simulator that replays last week's task traffic. We run three environments: dev, staging, and lab. The lab environment includes the GameBoy Advance and a spare Dell switch. If the WiFi signal drops, the router buffers task packets in a custom eBPF map. If the buffer fills, it writes the event to a local SQLite database and retries over a 4G modem. We call this the last-mile task tarpit. None of this is on the critical path, because Temporal workflows are durable.
Results and next steps¶
After rollout, cross-team handoffs became visible in one place. The GameBoy Advance terminal reduced the time to mark a lab task complete from several minutes to one button press. The router's electricity-aware firewalling eliminated false completions during power tests. The IPv6 WiFi mesh gave every task event a stable address. The Dell and Cumulus Linux underlay carried the load without a single spanning-tree incident. Our next step is to attach a second GameBoy Advance for the platform team and add a Temporal workflow that rotates the task cartridge. We are also evaluating a small OLED display for the router so on-call can see the current energy envelope.
Comments 6
Skeptical SRE
I love Temporal, but using it to paper over three trackers and a spreadsheet feels like buying a jet to commute to the mailbox. What happens when the GameBoy cartridge dies and nobody remembers the deterministic workflow ID?
Temporal Fan
Deterministic IDs are derivable from project and task, so that part is not terrible. The GBA is the piece I would not put anywhere near the critical path.
Chadwick “Patch” McJanky Author
Fair. The GBA is an attestation terminal, not the source of truth. If it dies, the workflow waits and the fallback activity posts to the internal task API. We keep a spare cartridge in the lab.
Network Ned
The IPv6-only SSID concerns me more. DNS64/NAT64 for legacy services is a classic place for weird task-event failures.
Packet Purist
Declarative nftables that only allow IPv6 traffic carrying a task workflow header? What header is that, exactly? An IPv6 extension header? Custom encapsulation? This sounds like a packet simulator's dream and an on-call nightmare.
GBA Homebrew Dev
Finally someone treats the GBA as serious hardware. The link-cable-to-ESP32 bridge is neat. But pressing A to complete a task sounds like a great way to accidentally mark the wrong thing done when the ROM scrolls too slowly.
Lab Rat
We had a lab status GBA too. The A button got mashed every time someone bumped the table. We moved to a rotary encoder and never looked back.
Cloud Native Cynic
This post has everything: Temporal, EVPN-VXLAN, eBPF, SQLite, a 4G modem, and a GameBoy Advance. I cannot tell if this is a real architecture or a Kubernetes-generated haiku.
Chadwick “Patch” McJanky Author
It is a real architecture, and the haiku is in the commit message. The 4G modem only handles the last-mile task tarpit, not the main path.
Concerned Auditor
Electricity-aware firewalling based on charger draw is clever until the smart plug reports stale wattage and blocks completions during a normal workload. Do you have a bypass that is not just waiting for the router to feel better?
IPv6 Pragmatist
SLAAC plus EVPN-VXLAN plus per-pod /64s is a solid lab design. But 'blazingly fast' and 'router buffers task packets in a custom eBPF map' do not belong in the same sentence unless you enjoy debugging memory pressure at 3 a.m.