The symptom: morale telemetry was trapped in silos¶
At ShitOps, our weekly Agile development retrospectives kept surfacing the same issue: communication between teams was fading, and team morale was drifting. Engineers on the Fleet Experience team had rich tesla telemetry from test vehicles, while Workplace Ops had x11 session events from meeting-room displays. Nobody could see both streams together. A release captain would ask whether a morale dip correlated with a room display crash or a vehicle telemetry gap, and the answer required three Slack threads, two spreadsheets, and a ceremonial spreadsheet reconciliation. That was not acceptable for a platform company.
We decided to make this problem declarative. The goal was a single control plane that could ingest tesla telemetry, observe x11 display surfaces, train a morale model with TFX, and route recommendations through a microservice. We named it MoraleMesh.
The architecture: everything is a Custom Resource Definition¶
We introduced three Custom Resource Definitions. CommunicationIntent captures a team-to-team signal contract, TeslaSignalPolicy binds a vehicle telemetry stream to a morale question, and X11Surface declares a meeting-room display as a first-class rendering target. A Kubernetes operator watches these resources and reconciles the world. Teams no longer file tickets to ask for a dashboard; they apply a CRD and wait for the operator to converge.
This is where the magic starts. Every CRD change emits an event to an Apache Beam pipeline. The pipeline enriches tesla telemetry with x11 surface metadata and writes both into a feature store. TFX then trains a morale trajectory model on the joined features. The trained model is promoted by a microservice called morale-inference-svc, which exposes a gRPC endpoint for the X11 renderer. The renderer paints a room-level morale pulse directly onto the meeting-room display.
The operator also schedules Agile development ceremonies. It reads the team morale model output and creates a CeremonyRecommendation event before standup. If morale is predicted to dip, the operator can inject a five-minute appreciation round into the calendar. This is not automation for its own sake; it is closed-loop team experience.
Why the X11 layer matters¶
Most dashboards stop at the browser. We needed to go deeper. Meeting-room displays run x11, and x11 surfaces expose window focus, idle time, and session lifecycle events. By modeling these as X11Surface resources, we can map a morale pulse to a specific physical room. The renderer subscribes to morale-inference-svc and uses x11 window properties to decide which surface gets the orange morale warning. A Tesla telemetry surge from a nearby test drive can warm the room's morale color before the standup even starts.
We use a sidecar to translate x11 events into CloudEvents. The sidecar is deployed with every meeting-room display and includes a small cache for offline resilience. If the API server is unavailable, the sidecar queues x11 events locally and replays them when the cluster returns. This keeps communication between teams uninterrupted even during control-plane upgrades.
The Apache Beam and TFX loop¶
The Apache Beam pipeline runs in streaming mode. It reads from a Kafka topic called tesla-signal-raw and a Pub/Sub topic called x11-surface-events. The pipeline windows both streams into five-minute intervals, joins them by room ID, and computes a morale_pressure feature. It also emits a backfill dataset for TFX. The TFX pipeline has four components: ExampleGen, Transform, Trainer, and Pusher. Every successful run produces a new model version and a signed model card.
To make this reliable, we retrain on every CommunicationIntent generation bump. A generation bump happens whenever a team updates its intent, which in Agile development can be weekly. This ensures the model always reflects the current communication between teams. The trade-off is more training runs, but the platform team has a dashboard for that.
The microservice and the morale API¶
morale-inference-svc is a Go microservice with a gRPC API. It caches model versions and exposes a GetRoomMorale method. The service has a readiness probe that checks the TFX model registry, a liveness probe that checks the Beam checkpoint, and a startup probe that checks the operator's CRD schema. If any probe fails, Kubernetes restarts the pod. That is exactly the kind of resilience a morale system needs.
We also added a MoraleMesh namespace and a set of Helm charts. The charts install the operator, the Beam pipeline, the TFX runner, and the x11 renderer. A single make install sets up the whole loop. New teams can join by applying a CommunicationIntent and a TeslaSignalPolicy. The onboarding guide has seven steps and a glossary.
Results and next steps¶
The first pilot showed a measurable improvement in team morale surveys, though we are still tuning the model. Communication between teams now happens through reviewable CRDs instead of ad hoc Slack threads. The X11 renderer has become a talking point in every office tour, and the tesla telemetry feed is finally visible where decisions are made. Our next milestone is a federated MoraleMesh that spans regions. We are also evaluating a second microservice for parking-lot sentiment and a TFX component for predicting whether a meeting should be an email. With Custom Resource Definitions, Apache Beam, and TFX working together, the platform is ready for the next phase of Agile development at ShitOps.
Comments 5
Kelsey Trautman
I appreciate the ambition, but treating X11 as a first-class rendering target feels like swapping browser sandboxing for a 1990s display-server attack surface. How are you isolating the sidecars and authenticating the x11 event stream before it becomes morale telemetry?
Daphne YAML-O'Connor Author
Fair concern. The sidecar runs with a per-room service account, only publishes a narrow CloudEvent schema, and the renderer never accepts arbitrary X11 commands from the network. We are also adding mTLS between sidecar and operator before the next pilot. The current post probably undersells that hardening.
Raj Patel
The per-room service account helps, but what happens when a display is physically compromised? Can it forge a morale pulse for another room, or are events room-scoped at the API server too?
Marcus Chen
The CRD model is elegant until every team has ten CommunicationIntent generations and nobody knows which bump triggered the retrain that changed the morale model. What does rollback look like when a weekly intent edit silently shifts the recommendations?
Nina Okafor
This is my main worry too. A generation bump causing retraining is a surprising side effect for what looks like a configuration object. I would want model lineage tied directly to the CRD UID and generation in the model card.
Priya Nair
Morale telemetry from Tesla vehicles plus room focus/idle events sounds like a privacy review waiting to happen. Are the features aggregated at room level, and what consent or notice do people get before their meeting-room behavior influences a morale model?
Tomás Rivera
This reads like a Rube Goldberg machine dressed as a control plane. A dashboard and a Slack bot could answer the original question with far fewer moving parts. The engineering is impressive, but is the complexity buying better decisions or just more surfaces to operate?
Avery Brooks
The probe chain concerns me: readiness checks TFX registry, liveness checks Beam checkpoint, startup checks CRD schema. If any dependency blips, morale-inference-svc restarts. That feels like it could amplify an outage rather than heal it. Have you tested partial dependency failures?