Building an AI Printer State machine for Self-driving cars with MQTT, rclcpp, Jaeger, SAP, discord, and icinga2

Fair challenge, and honestly we debated it. The regulatory picture in two of our markets currently requires a physical receipt for certain fare classes, and rewriting the compliance story was slower than making the printer observable. So we treated it as a billing boundary, not a product feature. Our longer-term plan is to make the receipt a logical artifact and treat the printer as one of several possible sinks. That's exactly why we're streaming printer telemetry into the same rclcpp graph as perception in phase two. Once the event model is unified, dropping the printer becomes a config change instead of a rewrite.

I kind of agree with Praxis but I think the point is subtler. The printer is almost incidental here; what they've really built is a generic "did this side effect actually happen" framework. Swap thermal printer for payment terminal or door lock and the same state machine applies. I'd love to see that abstraction named explicitly in the post.

Great questions. We run QoS 1 on state and decision, but QoS 2 on receipt specifically because the receipt topic is the one that crosses the billing boundary. Sessions are persistent for the bridge worker so it can catch up after a partition, which is also how icinga2 caught the two broker partitions mentioned in the rollout section. Quarantined receipts currently have a 72-hour TTL enforced by the replay worker. Anything older requires an explicit escalation because we don't want receipts reconciled into the wrong billing period. That policy is enforced in config rather than in code, which has already saved us once.

That makes sense, thanks for the detail. The 72-hour TTL tied to billing periods is a nice touch — I've seen reconciliation jobs silently cross period boundaries and create genuine accounting problems. Would love a follow-up post on the replay worker itself.

On alert fatigue: unresolved discord threads feed icinga2 as an aggregated check, not per-thread. A single alert fires when the unresolved thread count for a region crosses a threshold, and the check message links to the filtered discord view rather than opening a new incident per receipt. That kept the on-call rotation sane during the first rollout wave. On certs: fair point, and it's the weakest part of the current design. We're on 90-day rotation with automated issuance, but static IP plus long-lived certs is a coupling we're actively unwinding. Phase two moves to short-lived certs with identity derived from the vehicle attestation rather than the subnet. I'd rather flag that openly than pretend it's solved.

This is a genuinely sharp catch and you're right to call it out. In the current implementation the scorer reads the current window, which means replay can produce a different confidence score than the original evaluation. We handle that by storing the original score and feature snapshot on the decision event, so the replay path reuses the snapshot rather than re-scoring. That preserves reproducibility for audits, but it means the model never gets a second look at a receipt it misjudged. We're discussing a deliberate re-score mode as a separate, clearly-labelled path. It's an open design question and I should have raised it in the post.

Storing the feature snapshot on the decision event is the right call. A labelled re-score mode sounds like exactly the kind of thing that should be a separate state rather than a flag on the existing one — otherwise you lose the ability to tell an audit which path a receipt went through.

This sub-thread is the most useful thing in the comments section. Model reproducibility on replay is a problem that hits every event-driven ML system, not just printers. Would read a whole post on just this.

You're not wrong, and I'd rather own that than overclaim. The AI scorer's real contribution is the confidence banding — routing the obvious cases away from humans entirely and surfacing the genuinely ambiguous ones. The 917 auto-recovered receipts would otherwise have all landed in a support queue. But yes, a meaningful share of the remaining cases were hardware, and the discord button exists precisely because "refill the paper" doesn't need a model. If the post reads as AI-washed, that's on my framing. The honest summary is: instrumentation first, triage second, model third.

We reconcile from the SAP side on a scheduled sweep. If a receipt ID exists in our event store but has no corresponding SAP document number after the expected window, a reconciliation job queries SAP by receipt ID and backfills the decision topic. It's a boring batch job that exists purely to close the gap you're describing, and it's saved us more than once during broker partitions. The key detail is that the receipt ID is generated at the printer_sentinel node before the first MQTT publish, so the entire pipeline — bridge, scorer, adapter, reconciliation sweep — shares one immutable key. That's what makes idempotency actually work end to end.

The immutable key generated before first publish is the whole ballgame. Everything downstream can be retried, replayed, or reconciled as long as that key never changes. Thanks for spelling it out.

We did consider it, and honestly the rclcpp component is the most tightly coupled part of the design. The vendor SDK is not thread-safe and it's callback-driven, so wrapping it in a single node with a serialized event loop was the pragmatic choice for phase one. Splitting transport out would have meant another process and another failure mode on constrained vehicle hardware. That said, the Jaeger annotation is deliberately kept at the State machine transition layer rather than inside the SDK wrapper, so a vendor crash produces a Faulted transition with a trace span before the node restarts. It's not a clean separation, but it's a survivable one. Phase two's unified rclcpp graph is partly about untangling exactly this.

They were tuned against a labelled set of about 40,000 historical print jobs, where "labelled" means we manually verified whether the receipt physically existed after the fact. The 0.93 cutoff was chosen to keep false-billable below a threshold our finance team could tolerate, and 0.80 was chosen to keep the human review queue tractable. They're not magic numbers, but they're also not permanent. The bands are config values and we revisit them whenever the model is retrained. If you take one thing from the post, I'd rather it be the idea of explicit confidence bands with owned thresholds than the specific numbers.

You're right that the human layer scales worst, which is why the AI scorer's routing is the real scaling lever. At ten times the fleet, the expectation is that the confidence bands move further apart and the human review queue grows sub-linearly, because most new cases fall into the confident bands. If that assumption breaks, the discord bot becomes the bottleneck and we'd need tiered triage by region. We haven't had to build that yet, but I'd rather say it's unproven than claim it's solved.

It absolutely requires a confirmation, and the confirmation includes the receipt ID, the original trace link, and the billing period it will land in. We added the period display after someone nearly replayed a Q3 receipt in early Q4. Lesson learned the mildly expensive way.

The diagram simplifies a real intermediate step. ticket_ack moves the receipt into a re-evaluation queue, not straight to SAPReconcile. The operator's acknowledgement indicates the physical issue is resolved, and then the scorer re-runs against the stored feature snapshot. If it clears the billable threshold, it proceeds; otherwise it stays quarantined. The diagram collapses that for readability, which I now realize makes it look like a human click is sufficient confidence on its own. I'll update it to show the re-evaluation state explicitly — thanks for flagging it.

Thank you, genuinely. The silent-success failure mode is the reason we wrote this up at all. The most expensive incidents we've had weren't crashes, they were systems confidently reporting that everything was fine. Glad the quarantine rule is useful to you.