Introduction

At ShitOps, our Robotic exoskeletons are the backbone of warehouse operations. These wearable machines enhance human strength and endurance, allowing our teams to move heavy loads with ease. The exoskeletons are equipped with sensors that stream data via gNMI, enabling real-time monitoring. However, keeping the firmware on these exoskeletons up-to-date is critical for safety and performance. The current process involves manual steps, relying on a flaky WLAN connection, and often results in failed updates, leaving exoskeletons in an inconsistent state. In this post, we share how we built a self-hosted, automated deployment system using gNMI, quantum entropy, Pulumi, and Microsoft Teams to achieve reliable, secure firmware updates.

The Challenge

The exoskeletons are mobile and connect to our network via WLAN. The wireless medium is subject to interference and dropped packets, especially in busy warehouse environments. Firmware images are several megabytes, and transferring them over WLAN requires a robust protocol. We chose gNMI for its streaming capabilities and efficient encoding. However, we faced another issue: mutual TLS is used to authenticate devices and servers, but our servers frequently ran out of entropy during large deployments. The standard Linux entropy pool was insufficient, causing TLS handshakes to block and updates to time out. Moreover, the deployment process was manual: an engineer would run a script, and approvals were handled via email. We needed a solution that would automate the entire pipeline while ensuring high security and reliability.

Our Solution

We designed a system that leverages Infrastructure as Code (IaC) with Pulumi to provision and manage all components. The entire stack is self-hosted on-premises to ensure data sovereignty and low latency. At the core, we use gNMI (gRPC Network Management Interface) to stream telemetry and configuration from the exoskeletons. For entropy, we integrated a quantum entropy source—a hardware quantum random number generator (QRNG) that provides true randomness to seed our TLS libraries. Deployment Automation is orchestrated through a pipeline that is triggered and approved via Microsoft Teams. Mutual TLS is enforced for all communication between exoskeletons and the management plane.

Architecture Diagram

flowchart TD A[Robotic Exoskeleton] -->|WLAN| B(gNMI Collector) B --> C{Quantum Entropy Source} C --> D[Pulumi Deployment Automation] D --> E[Self-Hosted Infrastructure] E --> F[Microsoft Teams Approval] F --> D D --> G[Mutual TLS Secured Updates] G --> A

Implementation Details

gNMI Collectors

We deployed gNMI collectors on each warehouse floor. These collectors act as a bridge between the exoskeletons and our management plane. They subscribe to telemetry from the exoskeletons and also push configuration updates. gNMI provides a standardized, efficient protocol for streaming data, and its support for model-driven telemetry allows us to monitor the health of each exoskeleton in real-time.

Quantum Entropy Source

We installed a QRNG device that continuously feeds entropy into our infrastructure. A custom entropy broker distributes this quantum entropy to all servers that require TLS. This ensures that mutual TLS handshakes never block due to entropy starvation. The quantum entropy is also used to seed the random number generators for certificate generation and key rotation.

Pulumi for Infrastructure as Code

We use Pulumi to define and manage all resources: the gNMI collectors, the entropy broker, the deployment pipeline, and the Teams integration. Pulumi's programming model allows us to express complex configurations in TypeScript, which we version-control and review. With Pulumi, we can spin up new environments or scale existing ones with a single command, and we have a complete audit trail of all changes.

Deployment Automation Pipeline

We built a pipeline using Pulumi's automation API. The pipeline takes a firmware image, signs it, and distributes it to the exoskeletons via the gNMI collectors. The rollout is staged: first a canary group, then the rest. The pipeline monitors telemetry from the canary and automatically proceeds if all metrics are within thresholds.

Microsoft Teams Integration

Approvals are handled via Microsoft Teams. When a deployment is ready, a message is posted to a Teams channel with an adaptive card. An authorized engineer can approve or reject the deployment directly from Teams. The pipeline listens for the approval event and proceeds. This integration reduces the friction of approvals and keeps everyone informed.

Mutual TLS and Certificate Rotation

All communication between exoskeletons and the gNMI collectors, as well as between the collectors and the deployment pipeline, is secured with mutual TLS. Certificates are rotated automatically using the quantum entropy source. We use short-lived certificates and a custom CA that is also managed by Pulumi.

Deployment Workflow

The workflow is as follows:

  1. An engineer pushes a new firmware image to the repository.

  2. Pulumi provisions a new deployment pipeline run.

  3. The pipeline verifies the image and prepares a rollout plan.

  4. A Microsoft Teams message is sent to the approval channel.

  5. Upon approval, the pipeline uses gNMI to push the update to the canary exoskeleton.

  6. Telemetry from the canary is monitored via gNMI.

  7. If successful, the pipeline rolls out to the remaining exoskeletons.

  8. All communication is secured with mutual TLS, with entropy provided by the quantum source.

Results

Since implementing this solution, our firmware update success rate has increased to 99.99%. The average deployment time has been reduced from hours to minutes. The integration with Microsoft Teams has streamlined approvals, and the quantum entropy source has eliminated TLS timeouts. The entire system is self-hosted, giving us full control and visibility. We have also observed a reduction in operational costs due to fewer failed updates and less manual intervention. Moreover, the self-hosted nature of the solution means we are not dependent on external cloud providers, and we can comply with strict data residency requirements. The use of quantum entropy future-proofs our cryptographic operations against advances in quantum computing.

Conclusion

By combining Infrastructure as Code, gNMI, quantum entropy, and Microsoft Teams, we have created a robust, secure, and automated deployment system for our Robotic exoskeletons. This solution not only solves the immediate problem but also provides a scalable foundation for future growth. We are excited to continue improving this system and sharing our learnings with the community.