The problem: windows 8 kiosks, Matrix Space, and low latency

ShitOps operates a fleet of windows 8 kiosks in fulfillment centers. Each kiosk hosts a dispatch console that writes status updates into a Matrix Space used by field tech and operations. The kiosks also need to read inventory attestations anchored in Hyperledger. Last quarter we measured p95 update propagation at 4.2 seconds. The root cause was a legacy gateway that polled every 15 seconds, batched messages, and forwarded them to Matrix only after a scheduled job completed. In a low latency workflow, that delay is unacceptable. We decided to rebuild the path around EVPN, infrastructure as code, and a ledger-backed event mesh. The target was sub-100 ms propagation from kiosk to Matrix Space, with a verifiable audit trail for every dispatch event.

The architecture

flowchart TD A[windows 8 kiosk agent] -->|mTLS protobuf| B[Edge relay] B --> C[EVPN VXLAN fabric] C --> D[Matrix Space bridge] D --> E[Hyperledger Fabric chaincode] E --> F[Audit ledger] D --> G[Low latency event bus] G --> H[Ops dashboard] B --> I[Local Redis cache] I --> G

Every kiosk runs a signed agent. The agent publishes protobuf events over mTLS to the nearest edge relay. Edge relays are deployed on a three-tier EVPN fabric, so east-west traffic is never hairpinned through a central data center. The fabric advertises service routes with BGP EVPN. We treat all fabric, relay, Matrix bridge, and Hyperledger components as infrastructure as code. Terraform provisions the EVPN underlay, Crossplane reconciles Kubernetes resources, and Ansible configures the windows 8 scheduled tasks. The Matrix Space bridge consumes events from a low latency event bus and writes messages into the correct room. A Hyperledger Fabric chaincode records the digest of every event. Audit can prove that a dispatch message existed before it appeared in Matrix Space.

The event path

The windows 8 agent uses a signed protobuf envelope with a monotonic sequence number and a device attestation token. The edge relay validates the token, checks the sequence window in Redis, and forwards the event to a Kafka topic. Kafka is not used as a queue. It is used as a replayable ordering layer for the Matrix Space bridge and the Hyperledger writer. The bridge projects events into Matrix rooms and adds a custom transaction ID. The Hyperledger writer commits the event digest, the Matrix transaction ID, and the EVPN path tag. If the ledger commit fails, the bridge holds the Matrix message in a reconciliation queue and retries with exponential backoff. The dashboard shows bridge lag, ledger commit lag, and EVPN path health.

Infrastructure as code pipeline

We manage the entire topology with infrastructure as code. The repository has modules for EVPN underlay, relay fleets, Matrix homeserver configuration, Hyperledger orderers, and windows 8 task definitions. A pull request runs Terraform plan, Crossplane dry-run, and a policy check. The policy check rejects any change that does not include a rollback annotation. On merge, Argo CD applies the manifests to the management cluster. A controller then rolls the relay fleet one rack at a time. The windows 8 scheduled task is updated through an Ansible playbook with a canary group. We keep the entire state in a locked object store with versioning. This gives us repeatable infrastructure as code from edge to ledger.

Low latency controls

Low latency is a first-class requirement. We pin relay workers to NUMA nodes, use eBPF for service load balancing, and keep the bridge in the same EVPN tenant as the Matrix homeserver. We set TCP_NODELAY, use HTTP/2 keepalives, and run a sidecar that pre-opens TLS sessions. The Matrix Space bridge batches only within a 5 ms window. The event bus uses acks=1 with idempotent producers. The ledger writer uses async commit but a quorum read on reconciliation. We also run synthetic probes from each kiosk group. A probe writes a test event to a private Matrix Space and measures the round trip through EVPN, the bridge, and Hyperledger. If p95 exceeds 100 ms, the controller sheds non-critical inventory sync traffic to protect dispatch updates.

Rollout and results

We rolled out to 12 kiosks in the first wave. The windows 8 agents connected to the nearest relay over the EVPN fabric. The Matrix Space bridge wrote 48,000 events during the pilot. Hyperledger recorded 48,000 matching digests. The synthetic probes show p95 propagation of 87 ms at the edge and 94 ms at the dashboard. The audit team can now trace any dispatch message back to a device, an EVPN path tag, and a ledger block. The field tech team reports that status updates appear in the Matrix Space before the kiosk screen finishes its refresh animation. We are expanding to all 3,200 kiosks next quarter.

Lessons learned

The biggest lesson is that windows 8 devices need a predictable network contract. EVPN gives us that contract at the fabric layer. Infrastructure as code gives us repeatability from the kiosk task to the Hyperledger orderer. The Matrix Space bridge gives operations a familiar interface. The low latency event bus gives us a replayable path for reconciliation. Hyperledger gives audit a tamper-evident record without slowing down the main path. This design is now the reference pattern for every kiosk-to-Matrix workflow at ShitOps. We are also evaluating a second ledger channel for returns, a third relay tier for cold storage, and a WASM filter for event enrichment at the edge.