The symptom: morale telemetry was trapped in silos

At ShitOps, our weekly Agile development retrospectives kept surfacing the same issue: communication between teams was fading, and team morale was drifting. Engineers on the Fleet Experience team had rich tesla telemetry from test vehicles, while Workplace Ops had x11 session events from meeting-room displays. Nobody could see both streams together. A release captain would ask whether a morale dip correlated with a room display crash or a vehicle telemetry gap, and the answer required three Slack threads, two spreadsheets, and a ceremonial spreadsheet reconciliation. That was not acceptable for a platform company.

We decided to make this problem declarative. The goal was a single control plane that could ingest tesla telemetry, observe x11 display surfaces, train a morale model with TFX, and route recommendations through a microservice. We named it MoraleMesh.

The architecture: everything is a Custom Resource Definition

We introduced three Custom Resource Definitions. CommunicationIntent captures a team-to-team signal contract, TeslaSignalPolicy binds a vehicle telemetry stream to a morale question, and X11Surface declares a meeting-room display as a first-class rendering target. A Kubernetes operator watches these resources and reconciles the world. Teams no longer file tickets to ask for a dashboard; they apply a CRD and wait for the operator to converge.

This is where the magic starts. Every CRD change emits an event to an Apache Beam pipeline. The pipeline enriches tesla telemetry with x11 surface metadata and writes both into a feature store. TFX then trains a morale trajectory model on the joined features. The trained model is promoted by a microservice called morale-inference-svc, which exposes a gRPC endpoint for the X11 renderer. The renderer paints a room-level morale pulse directly onto the meeting-room display.

flowchart LR A[CommunicationIntent CRD] --> B[Kubernetes API] C[TeslaSignalPolicy CRD] --> B D[X11Surface CRD] --> B B --> E[MoraleMesh Operator] E --> F[Apache Beam Pipeline] G[tesla telemetry] --> F H[x11 session events] --> F F --> I[Feature Store] I --> J[TFX Trainer] J --> K[morale_inference_svc] K --> L[X11 Renderer] L --> M[Meeting Room Display] E --> N[Agile Ceremony Scheduler]

The operator also schedules Agile development ceremonies. It reads the team morale model output and creates a CeremonyRecommendation event before standup. If morale is predicted to dip, the operator can inject a five-minute appreciation round into the calendar. This is not automation for its own sake; it is closed-loop team experience.

Why the X11 layer matters

Most dashboards stop at the browser. We needed to go deeper. Meeting-room displays run x11, and x11 surfaces expose window focus, idle time, and session lifecycle events. By modeling these as X11Surface resources, we can map a morale pulse to a specific physical room. The renderer subscribes to morale-inference-svc and uses x11 window properties to decide which surface gets the orange morale warning. A Tesla telemetry surge from a nearby test drive can warm the room's morale color before the standup even starts.

We use a sidecar to translate x11 events into CloudEvents. The sidecar is deployed with every meeting-room display and includes a small cache for offline resilience. If the API server is unavailable, the sidecar queues x11 events locally and replays them when the cluster returns. This keeps communication between teams uninterrupted even during control-plane upgrades.

The Apache Beam and TFX loop

The Apache Beam pipeline runs in streaming mode. It reads from a Kafka topic called tesla-signal-raw and a Pub/Sub topic called x11-surface-events. The pipeline windows both streams into five-minute intervals, joins them by room ID, and computes a morale_pressure feature. It also emits a backfill dataset for TFX. The TFX pipeline has four components: ExampleGen, Transform, Trainer, and Pusher. Every successful run produces a new model version and a signed model card.

To make this reliable, we retrain on every CommunicationIntent generation bump. A generation bump happens whenever a team updates its intent, which in Agile development can be weekly. This ensures the model always reflects the current communication between teams. The trade-off is more training runs, but the platform team has a dashboard for that.

The microservice and the morale API

morale-inference-svc is a Go microservice with a gRPC API. It caches model versions and exposes a GetRoomMorale method. The service has a readiness probe that checks the TFX model registry, a liveness probe that checks the Beam checkpoint, and a startup probe that checks the operator's CRD schema. If any probe fails, Kubernetes restarts the pod. That is exactly the kind of resilience a morale system needs.

We also added a MoraleMesh namespace and a set of Helm charts. The charts install the operator, the Beam pipeline, the TFX runner, and the x11 renderer. A single make install sets up the whole loop. New teams can join by applying a CommunicationIntent and a TeslaSignalPolicy. The onboarding guide has seven steps and a glossary.

Results and next steps

The first pilot showed a measurable improvement in team morale surveys, though we are still tuning the model. Communication between teams now happens through reviewable CRDs instead of ad hoc Slack threads. The X11 renderer has become a talking point in every office tour, and the tesla telemetry feed is finally visible where decisions are made. Our next milestone is a federated MoraleMesh that spans regions. We are also evaluating a second microservice for parking-lot sentiment and a TFX component for predicting whether a meeting should be an email. With Custom Resource Definitions, Apache Beam, and TFX working together, the platform is ready for the next phase of Agile development at ShitOps.