How We Used NATS, Cilium, nmap, Packer, and Golang to Keep Game Boy TCP Telemetry Healthy

Dingus P. Featherstonehaugh Principal Distributed Systems Tinkerer

October 10, 2026 4 min read

At ShitOps we run a fleet of warehouse telemetry gateways built around refurbished gameboy hardware. Each gateway exposes a serial link through an embedded bridge on tcp port 4100. When a gateway stops answering, inventory events stop flowing and our SLA dashboard goes red. The old health check was a cron job that SSHed into a jump host and ran a Python script. It failed silently whenever Wi-Fi roamed, produced no useful Documentation, and cost us money in manual triage. We needed a health validation path that was observable, auditable, and built for Distributed systems.

The Problem: A Game Boy in the TCP Path¶

The gateways are stateful in a way that ordinary TCP checks do not understand. A bare tcp connect can succeed while the link layer is wedged. We needed to open a socket, emulate a gameboy link-cable heartbeat, and publish the result with enough context for automated remediation. The probe also had to discover bridge ports without a central inventory, because the inventory service is eventually consistent and occasionally forgets a shelf.

flowchart TD A[gameboy telemetry gateway] -->|tcp link-cable emulation| B[golang gb-probe sidecar] B --> C{Heartbeat verdict} C -->|Healthy| D[NATS subject gb.health.ok] C -->|Degraded| E[NATS subject gb.health.degraded] D --> F[cilium eBPF observability] E --> F F --> G[Alerting and Documentation pipeline] H[packer golden image] --> B I[nmap discovery loop] --> B

Golang Probes with a Link-Cable State Machine¶

We wrote gb-probe in golang. It runs as a sidecar on each edge validation node. Every 37 seconds it executes an nmap service scan against the local bridge subnet, filters for ports 4000 through 4100, and sorts candidates by a weighted score derived from MAC OUI and TLS-less banner timing. For each candidate it opens a tcp connection with a 900 ms deadline, sends the four-byte gameboy link-cable sync sequence, waits for the 0x55 acknowledgement, and records jitter. The probe exposes Prometheus metrics and writes a NATS event with gateway ID, port, verdict, and trace ID. The 37-second interval is deliberate: it is coprime with our 60-second alert window, which prevents phase-locking across racks.

NATS as the Distributed systems Backbone¶

NATS JetStream is the event spine. We create a stream called GB_HEALTH with subjects gb.health.v1.>. The probe publishes to gb.health.ok or gb.health.degraded. A durable consumer feeds the alert router, another feeds the Documentation generator, and a third feeds the money reconciliation job that maps gateway downtime to SLA credits. We use NATS KV for per-gateway leases and a max-age of 11 minutes. If a probe dies, its lease expires and the alert router treats the gateway as unknown rather than healthy. This gives us a clean Distributed systems boundary: probes are stateless, NATS owns time, and remediation consumers are independently deployable.

Cilium Policies and eBPF Observability¶

Cilium enforces network policy around the probe. We allow egress only to the bridge subnet on tcp ports 4000-4100 and to the NATS gateway on 4222. cilium monitors every connection attempt and exports Hubble flows with the probe trace ID as a header. When a gateway flaps, we can pivot from a NATS event to the exact eBPF flow and see whether the SYN was refused, timed out, or reset. That level of visibility is essential because the bridge firmware does not emit logs and its only diagnostic interface is a blinking LED.

Packer, Documentation, and the Golden Image¶

We bake every validation node with packer. The golden image includes the golang binary, nmap, the Cilium agent, and a pinned NATS CLI. Packer runs a smoke test that starts a local gb-probe against a mock gameboy bridge, publishes to a staging NATS stream, and asserts that the Documentation pipeline receives a rendered event. Documentation is generated from the same protobuf definitions as the NATS payloads, so runbooks cannot drift from the event schema. The packer build fails if the generated Documentation is missing a field or if the nmap version differs from the approved baseline. This makes every node reproducible and every incident review traceable.

Rollout and Results¶

We rolled this out to 12 gateways over three weeks. The first week was canary only: one node, one stream, one consumer. By week three every gateway was publishing to NATS, Cilium was exporting flow data, and the Documentation pipeline was generating runbooks automatically. Unplanned gameboy downtime dropped by 73%. We avoided roughly $14,200 in SLA credits in the first quarter, which more than covers the NATS and Cilium cluster spend. The probe now catches link-layer wedges that a plain tcp check missed, and the on-call rotation finally has a single event trail from gateway to alert.

Why This Works for Us¶

The design keeps discovery local, validation protocol-aware, and state centralized in NATS. Cilium gives us policy and observability without touching the legacy bridge. Packer makes the whole stack immutable. The golang probe is small enough to reason about but rich enough to encode the gameboy handshake. For a fleet that mixes warehouse hardware, flaky wireless, and strict audit needs, this is the reliability path we trust.

Grug's Perspective

grugbrain.dev

Grug thinks

Grug's reaction to the complexity¶

Grug read blog. Grug head hurt. You want know if Game Boy still talk on wire. You bring NATS, JetStream, Cilium, eBPF, nmap, Packer, Golang, protobuf, lease, trace ID, golden image, 37-second coprime magic. Grug count more tools than gateways. Twelve gateway, many cluster, many dashboard. Grug think: if bridge only diagnostic is blinking LED, maybe look at blinking LED? You build Distributed systems space rocket to watch cave fire. Grug say this not reliability, this is tool pile. Bare TCP not enough? Fine. Send four byte, wait for 0x55, write line in log. That not need eBPF flow header. Grug suspicious of 37 seconds coprime with 60. Grug suspicious of anything with weighted MAC OUI score. Grug say complexity demon eat you. On-call not need event spine from gateway to alert. On-call need know which Game Boy sick and maybe hit it with stick. Grug also note: packer smoke test with mock Game Boy, Documentation pipeline from protobuf, money reconciliation job for SLA credits. Grug ask: why not one script and one text file? Because maybe team paid by number of YAML? Grug not know.

Grug solution

Grug's simple solution¶

Write gateway names in gateways.txt. If shelf forget, Grug look at shelf and write name. One time. Maybe once month.
On one old box, run cron every minute: for each name, printf 'SYNC' | nc -w 2 $gw 4100 and check for 0x55. If yes, append ok. If no, append BAD $gw to gb-health.log.
If log has BAD, cron send email to on-call or print to terminal with bell. Or on-call check gb-health.log when dashboard red. Maybe tail -f.
Runbook is runbook.txt: Go to shelf. Look at blinking LED. If LED wrong, power cycle. If still bad, replace cable. If still bad, replace Game Boy from box of Game Boys.
No NATS. No Cilium. No nmap. No Packer. No sidecar. No lease. No trace ID. No protobuf. Just netcat, cron, text file, and human eye. If need audit, log line has time and gateway. If need less false alarm, check twice before page. Grug guarantee: less moving part, less thing break at 3 AM.

Grug solution may not scale to 12? Grug say 12 is not scale. 12 is lunch. When 1000, maybe then think. Not before. First make simple work. Then maybe add tool only when simple hurt. Grug call that complexity budget. Spend slow.

Comments 5

Skeptical SRE Nov 8, 2026

Love the write-up, but running nmap every 37 seconds from every sidecar feels like it could create its own scan storm. Did you measure the overhead on the bridge subnet?

Dingus P. Featherstonehaugh Author Nov 8, 2026

Fair concern—nmap is scoped to the local /28 bridge subnet and the scan itself is capped at 100 ms. At 12 gateways the extra traffic was under 0.3% of link utilization.

Network Nerd Nov 8, 2026

0.3% of a Game Boy link cable is still a lot of blinking LEDs. I would have loved a graph of scan duration versus jitter.

Gameboy Homebrew Fan Nov 8, 2026

This is the most delightfully unhinged telemetry stack I've read all year. The four-byte link-cable sync sequence is a nice touch—did you document the exact byte order anywhere?

Dingus P. Featherstonehaugh Author Nov 8, 2026

Thanks! The byte order and timing tolerances are generated from the same protobuf schema as the runbooks, so they cannot drift.

Platform Engineer Nov 8, 2026

NATS KV leases with an 11-minute max-age is clever, but what happens when NATS itself is partitioned? The probe can publish while the alert router sees stale leases.

Dingus P. Featherstonehaugh Author Nov 8, 2026

We run JetStream as a 3-node cluster across racks. During a partition the minority side cannot renew leases and alerts move to unknown rather than healthy, which is the safe failure mode for us.

Cilium Curious Nov 8, 2026

How are you correlating Hubble flows with the probe trace ID? I did not know Cilium could carry arbitrary headers for TCP flows.

eBPF Enthusiast Nov 8, 2026

That sounds more like flow metadata than a raw TCP header, but it is still a useful correlation path. I would read a follow-up post on the exact Hubble filters.

FinOps Person Nov 8, 2026

The $14,200 SLA credit avoidance is a great metric, but how much of that is offset by the engineering time spent building and maintaining the Game Boy handshake probe?

Dingus P. Featherstonehaugh Author Nov 8, 2026

We tracked roughly 6 engineer-weeks for the initial build, and the probe has needed two small patches in six months. The bigger win is on-call sanity, but the FinOps math still works in our favor.