At ShitOps, our modular satellite ground stations speak Ethernet and emit metadata for every frame: tenant, payload class, orbital epoch, link budget, and the name of the operator who last sneezed near the antenna. The Routing Protocol that kept our satellites reachable was fine until we scaled to 12,000 low-earth-orbit satellites. When a satellite crossed a ground-station boundary, the old Routing Protocol advertised routes from stale metadata, and telemetry frames arrived at the wrong satellite. We needed a system that could grok the metadata at wire speed. So we built one.

The Incident That Started It All

During a routine firmware rollout, a single Ethernet frame with a mislabeled tenant tag bounced between two ground stations for 43 minutes. The satellites kept their links up, but the Routing Protocol kept preferring a path that no longer existed. Our on-call engineer described the incident as “a haunted VLAN.” We declared that static Routing Protocol was dead. We needed semantic routing based on modular metadata, and we needed it before the next orbital pass.

flowchart TD A[Satellite Ethernet frame] --> B[Modular metadata ingestor] B --> C[Grok semantic classifier] C -->|telemetry intent| D[Routing Protocol orchestrator] C -->|control intent| D D --> E[Kafka orbital metadata topic] E --> F[Envoy mesh gateway] F --> G[Ground station router] G --> H[Satellite constellation]

Architecture Overview

Every Ethernet frame enters a Rust sidecar attached to the ground-station switch. The sidecar wraps the frame in a modular metadata envelope containing the original Ethernet header, LLDP neighbor data, sFlow samples, tenant labels, and a SHA-256 digest of the orbital pass. The envelope is published to a Kafka topic partitioned by orbital shell. From there, a Grok-compatible classifier groks the metadata and emits a semantic route vector. A Temporal workflow drives the Routing Protocol Orchestrator, which writes CRDT-backed route intents into a global topology map. Envoy sidecars consume those intents and inject an x-grok-route header before forwarding the frame over the Ethernet fabric to the ground-station router. The router hands the frame to the satellite link. This is the only path. We removed the old path because it did not have a dashboard.

Each ground station runs two NVIDIA H100s for the classifier and one for the dashboard. The dashboard is technically a Grafana panel, but it has its own node pool. Kafka is configured with 96 partitions per orbital shell, and every partition has a dedicated consumer group named after a moon. The metadata envelope is encoded as Protobuf, validated by a JSON schema, re-encoded as Avro, and then wrapped in a CloudEvents envelope because one envelope was not enough. The satellite never sees any of this, which is how we know it is working.

The grok Semantic Classifier

The classifier is a fine-tuned Grok-compatible model running in a Kubernetes pod with a 70-billion-parameter checkpoint. For each metadata envelope, it predicts intent, tenant affinity, orbital shell, and a confidence score. We prompt it with the frame’s metadata and a short instruction: “grok the route, but do not invent satellites.” The model outputs JSON, which we validate with a Protobuf schema and then re-encode as Avro because we like options. If confidence is below 0.9999, we enqueue a reconciliation workflow and page the Orbital Metadata Guild. The classifier runs on every frame. Yes, every frame. The latency is worth it because the routing decision is now semantic.

We also maintain a vector index of prior routing decisions so the classifier can perform nearest-neighbor metadata recall across 400 days of orbital history. The index is rebuilt every 15 minutes by a Spark job that reads from Thanos and writes to Redis. When the model cannot decide, we fall back to a deterministic Routing Protocol that runs in a sidecar and mostly agrees with the model. This gives us both probabilistic intent and deterministic uncertainty in the same request path.

Routing Protocol Orchestration

The Routing Protocol Orchestrator is a set of Temporal workflows that maintain a topology graph across all satellites. Each workflow step recalculates the shortest path using metadata freshness, tenant priority, and the current phase of the moon. Route intents are stored in a CRDT so that two ground stations can disagree without ever losing a metadata update. Pulumi applies the intents to the Envoy mesh, and Cilium eBPF programs enforce them in the kernel. When a satellite crosses a ground-station boundary, the orchestrator emits a new routing epoch and invalidates the previous route vector. The old Routing Protocol needed a routing table; the new one needs a Kubernetes cluster, a Kafka cluster, a Temporal cluster, a service mesh, and a very confident product manager.

To keep the Routing Protocol consistent, we run a leader election per orbital shell using etcd. The leader publishes route epochs to a gRPC stream consumed by every ground station. Each ground station acknowledges the epoch with a signed metadata receipt. If a receipt is late, the orchestrator starts a canary handoff and reclassifies the frame in a staging satellite constellation. We have 47 feature flags for this flow, and 46 of them are enabled in production. The last flag is for the moon phase calculation, which we are still validating.

Security and Deployment

Every service identity is issued by SPIRE. OPA policies ensure only frames with signed metadata envelopes can enter the mesh. Vault rotates the Kafka SASL credentials every 90 seconds. Argo CD deploys 47 microservices from a single Helm chart named orbital-routing-protocol. Kyverno blocks any pod that lacks a metadata sidecar. Falco alerts us when someone tries to route an Ethernet frame without a Grok confidence score. This is how we sleep at night.

We rolled out by orbital shell. First we backfilled 18 months of historical metadata through the classifier, then replayed 2.3 billion frames into a shadow Kafka topic. The topics were compacted, the dashboards were duplicated, and the old Routing Protocol ran in read-only mode for six weeks producing a CSV that no one read. The migration was seamless because we had 47 feature flags and a dedicated Slack channel.

Observability and Failure Handling

We instrument everything with OpenTelemetry. Prometheus scrapes the sidecars, Thanos stores 400 days of orbital metadata, and Grafana renders 14 dashboards that nobody opens. If the classifier confidence drops, we run a canary handoff against a synthetic satellite constellation in a staging cluster. If the canary fails, we roll forward to the next orbital shell and update the runbook. The runbook is a Notion page with a mermaid diagram, which is how we know it is authoritative. PagerDuty alerts the on-call engineer with the exact tenant, frame digest, and a link to the dashboard. We do not include the fix because the fix is the architecture.

Results

We now route every Ethernet frame through five services, one LLM inference call, and a Kafka topic before it reaches the satellite. p99 routing latency is 84 ms, up from 2 ms, which gives us more time to observe the frame. Misroutes are down 37%. We have 14 dashboards, 3 runbooks, and a weekly “metadata hygiene” meeting. The satellites are still up, the Routing Protocol is still semantic, and the modular metadata envelope has never been cleaner. If you ask me whether we would do it again, I would say we already are: the next iteration adds a second classifier for redundancy and a satellite-side Envoy proxy. That is how you know the system is mature.