Building a Multi-Tenant Event Streaming Fabric with Podman-Powered Distributed Proxy Pods

Gideon 'Giddy' P. Waffleberry Principal Tenant Fabric Architect

October 24, 2026 6 min read

The Incident That Started It All¶

At 09:42 on a Tuesday, ShitOps' quarterly tenant onboarding wave hit our event streaming platform. Tenant A's analytics workload began receiving a trickle of Tenant B's order lifecycle events. No data was lost, but our shared distributed proxy route table had reloaded while a tenant-specific listener was being registered. The proxy had been doing what we asked: forwarding event streams based on topic prefixes. The problem was that our prefixes were not authoritative across all tenants, and the reload path had no transactional boundary. We needed a tenant-aware event streaming fabric that could route every event through a distributed proxy with Podman-isolated data planes, without relying on a shared mutable route table.

flowchart TD A[Tenant Request] --> B[Edge Distributed Proxy Envoy + Wasm] B --> C{Tenant ID Present} C -- No --> D[Reject with 403] C -- Yes --> E[SPIFFE Identity Check] E --> F[OPA Policy Evaluation] F --> G[Podman Pod per Tenant] G --> H[Sidecar Event Stream Adapter] H --> I[Kafka Topic Namespace] H --> J[NATS JetStream Mirror] I --> K[Postgres Tenant Registry] J --> K K --> L[OpenTelemetry Collector] L --> M[Grafana Tempo]

Design Goals¶

We set four non-negotiable goals. First, every tenant gets a dedicated Podman pod boundary for its event streaming adapters. Second, every request traverses a distributed proxy that resolves tenant identity before any broker connection is opened. Third, all routing state is derived from a multi-tenant control plane with optimistic concurrency. Fourth, we can replay a tenant's event stream in a sandbox without touching production topics.

Architecture Overview¶

The resulting system, TenantMesh, is a multi-tenant event streaming fabric built around Podman pods and a distributed proxy tier. Each tenant receives a rootless Podman pod containing an event adapter, a policy sidecar, a metrics sidecar, and a local NATS JetStream leaf node. The edge distributed proxy is an Envoy fleet with WebAssembly filters for tenant extraction, SPIFFE/SPIRE identity verification, and OPA decision caching. Kafka remains the durable event streaming backbone; NATS JetStream acts as a tenant-local mirror for low-latency fan-out. A Postgres-backed tenant registry stores the canonical mapping of tenant to topic namespace, and a CRDT reconciler keeps proxy caches convergent across regions.

Podman-Powered Tenant Pods¶

Each tenant pod is declared as a Podman Quadlet unit and managed by systemd. We use rootless Podman with user namespaces, so each tenant's adapter runs under a distinct UID mapping. The adapter connects to Kafka with mTLS and to NATS with JWT-based credentials. A policy sidecar evaluates OPA bundles on every batch flush; a metrics sidecar exposes OpenTelemetry traces to the collector. Pods are placed on dedicated node pools labeled by tenant tier. Network isolation is enforced with netavark and per-pod firewall rules. We generate the Quadlet units from a Crossplane composition, which means tenant onboarding is a Git commit rather than a ticket.

Distributed Proxy Layer¶

The distributed proxy is the only ingress for tenant event streams. Every client connects to the nearest Envoy edge using HTTP/2 or gRPC. A Wasm filter extracts the tenant claim from the SPIFFE SVID, validates it against the tenant registry, and injects a routing header. OPA then evaluates a signed policy bundle: allowed topics, rate limits, schema version, and retention class. The proxy opens a connection to the tenant's Kafka namespace through a tenant-scoped upstream cluster. No shared route table exists. Route state is compiled into an immutable xDS snapshot and distributed with a versioned envelope. If a tenant changes, we publish a new snapshot; proxies keep serving the old snapshot until the new one is acknowledged by a quorum of edge nodes.

Event Streaming Backbone¶

Kafka provides durable, ordered event streaming. We create one topic namespace per tenant, with naming enforced by the proxy and by an ACL sync job. NATS JetStream mirrors selected subjects into the tenant pod for local fan-out and replay. A Temporal workflow orchestrates schema registration, ACL sync, mirror creation, and smoke tests. The workflow is idempotent and retries with exponential backoff. For cross-region tenants, a CRDT reconciler merges route deltas and resolves conflicts by policy version and timestamp. We also run a shadow consumer that verifies every produced event against the tenant schema before it is acknowledged upstream.

Multi-Tenant Control Plane¶

The control plane is a GraphQL federation gateway over tenant, policy, and stream services. It writes to Postgres with row-level security and emits change events to Kafka. A Dapr pub/sub component fans those changes out to reconcilers. Kyverno-style admission policies validate tenant manifests before they reach the reconciler. Crossplane compositions provision Podman pods, Envoy clusters, Kafka namespaces, and NATS mirrors. Every change is signed and audited. The multi-tenant registry stores tenant ID, SPIFFE trust domain, topic prefix, retention tier, and allowed egress. The registry is the source of truth; proxy caches are disposable.

Lifecycle and Reconciliation¶

Tenant onboarding starts with a pull request containing a TenantMesh manifest. CI runs schema compatibility checks, policy unit tests, and a dry-run reconciliation. Argo CD syncs the manifest to the control plane. Temporal executes the onboarding saga: create Kafka namespace, create NATS mirror, generate Podman Quadlet, deploy pod, warm proxy snapshot, run smoke test, enable traffic. Rollback is another saga. We keep three generations of proxy snapshots and can pin a tenant to a previous snapshot during an incident. The reconciler emits a heartbeat every 10 seconds; if a tenant pod misses three heartbeats, the proxy degrades that tenant to read-only and pages the on-call engineer.

Observability and SLOs¶

Every hop is instrumented with OpenTelemetry. The edge proxy emits span attributes for tenant ID, policy version, snapshot version, and decision latency. The event adapter emits batch size, flush duration, and broker error class. Grafana Tempo stores traces; Prometheus stores metrics; Loki stores structured logs. We define SLOs per tenant tier: p99 proxy decision latency under 40 ms, p99 end-to-end event delivery under 250 ms, and zero cross-tenant deliveries. A continuous canary tenant produces synthetic events every second and verifies isolation by attempting a forbidden topic read. The canary fails closed and triggers a snapshot rollback if isolation is breached.

Rollout Results¶

We rolled TenantMesh out to 42 tenants over six weeks. During the first canary week, the cross-tenant canary detected two policy cache misses; both were resolved by snapshot rollback before any production event was exposed. Proxy decision latency stayed under 31 ms at p99. Tenant onboarding time dropped from three days to eleven minutes. The Podman pod model gave us clean resource accounting per tenant, and the distributed proxy eliminated shared route table reloads. We now treat every tenant as a first-class event streaming boundary.

Lessons Learned¶

The biggest lesson was that identity must travel with the event. We now require SPIFFE SVIDs on every proxy hop and never trust a topic prefix alone. The second lesson was that route state should be immutable and versioned; xDS snapshots gave us deterministic rollbacks. The third lesson was that Podman Quadlets are an excellent fit for tenant-scoped systemd services. We are already planning TenantMesh v2 with per-tenant Wasm policy bundles, region-aware snapshot quorums, and a self-service replay console.

Grug's Perspective

grugbrain.dev

Grug thinks

Grug Reaction¶

Grug read post. Grug brain hurt. Many big words: multi-tenant event streaming fabric, Podman-powered distributed proxy pods, SPIFFE, OPA, Wasm, CRDT, xDS, Temporal, Crossplane, Dapr, Kyverno, GraphQL federation. Grug count many systems just to stop Tenant A see Tenant B order event. Grug suspicious. Problem not need fabric. Problem is prefix not authoritative and reload not atomic. Grug say: if event no carry tenant ID, no proxy can save. If reload not atomic, make reload atomic. Team build space rocket to cross small stream. Every new noun add one failure mode. Canary detect two cache misses? Grug ask: why cache? Route table small. Grug not trust magic snapshot quorum when simple ACL exist. Complexity demon got buffet. Podman pod per tenant, policy sidecar, metrics sidecar, NATS mirror, Kafka namespace, Postgres registry, CRDT reconciler, shadow consumer, heartbeat every 10 seconds, saga rollback. For 42 tenants? Grug tribe have 42 people maybe one cave. Onboarding in 11 minutes still too long for create topic. Grug laugh at v2 with more Wasm. Grug sad because on-call will page at 3am when Dapr pub/sub component hiccup. Grug say use boring thing.

Grug solution

Grug Solution¶

Put tenant_id in every event header and payload. Make one Kafka topic per tenant: tenant..orders. Set Kafka ACL so tenant only read and write own topic. Consumer group per tenant. No shared prefix routing. No proxy deciding tenant from prefix. If event carry tenant ID and topic ACL, Tenant A cannot see Tenant B. For route config: one small file or DB table tenant -> topic. Reload with write temp file then rename, or use Kafka Admin API. No shared mutable route table magic. Onboard tenant: create topic, set ACL, insert row, done. Canary: tiny script sends event with wrong tenant to wrong topic and expect deny. Replay: consumer group from offset. No Podman pod, no Wasm, no SPIFFE SVID dance, no CRDT, no Temporal saga, no shadow consumer. Grug fix bug in afternoon, go nap.

Comments 5

Priya Nair Nov 8, 2026

The immutable xDS snapshot approach is the part I trust most here. The canary catching policy cache misses before exposure is a strong validation. My question is about the quorum acknowledgement: when edge nodes are partitioned across regions, do you ever serve a stale snapshot for a tenant that has just been pinned to an older generation during an incident?

Gideon 'Giddy' P. Waffleberry Author Nov 8, 2026

Great question. We treat snapshot pinning as an explicit tenant-level override in the registry, and the proxy resolves that override before applying the quorum rule. During a partition, an edge keeps serving its last acknowledged snapshot, but if the tenant is pinned, the override is included in the signed envelope and propagates through the regional control plane as soon as connectivity returns. The important part is that we never mix snapshots across tenants; the envelope is versioned and tenant-scoped.

Marco Ruiz Nov 8, 2026

Does that mean a pinned tenant can stay on an old snapshot indefinitely if the region never reconnects? What is the guardrail against silent drift?

Gideon 'Giddy' P. Waffleberry Author Nov 8, 2026

There is a max pin duration and a drift alarm. The reconciler emits a heartbeat with the active snapshot generation, and if a pinned tenant exceeds the allowed window we alert and either extend the pin with an incident reference or roll forward. We also expose snapshot age as a per-tenant metric so drift is visible rather than silent.

DevOps Dave Nov 8, 2026

Rootless Podman plus systemd Quadlet units per tenant is a clean boundary, but it also sounds like a lot of systemd surface area. At 42 tenants it is manageable; at 500 tenants, who owns unit drift, failed restarts, and node pool placement? I would want to see the operational runbook before I bought into this.

Elena Sokolova Nov 8, 2026

This was my reaction too. Quadlets are great until you have hundreds of generated units and a few hand-edited exceptions. The Crossplane composition needs a strong no-drift policy and a reconciliation loop that can detect unit mutations.

Gideon 'Giddy' P. Waffleberry Author Nov 8, 2026

Fair concern. We generate everything from the composition and treat hand edits as drift. The reconciler compares the rendered Quadlet against the registry-derived desired state, and a mismatch alerts instead of silently overwriting unless the tenant is marked for emergency repair. Node placement is driven by tenant tier labels, and failed restarts are surfaced through the same heartbeat and SLO path as proxy health.

Security Sally Nov 8, 2026

SPIFFE SVIDs on every hop and OPA decision caching are the right direction. The part I want more detail on is the sidecar boundary: what stops a compromised metrics or policy sidecar in a tenant pod from reaching another tenant's NATS mirror or Kafka namespace? Rootless Podman helps, but it is not a complete tenant security boundary by itself.

Tomasz Krol Nov 8, 2026

I would also ask about the OPA bundle signing and rotation story. A cached policy decision is only as good as the bundle provenance and the cache invalidation path.

Kafka Karen Nov 8, 2026

I am not fully convinced by the NATS JetStream mirror. Kafka is already the durable backbone, and adding a tenant-local mirror gives you low-latency fan-out but also introduces dual-write ordering questions. How do you guarantee that a consumer reading from the NATS mirror sees the same order as the Kafka topic, especially during replay or schema migration?

Gideon 'Giddy' P. Waffleberry Author Nov 8, 2026

We do not treat NATS as a second source of truth. The mirror is a derived, tenant-local cache for selected subjects, and Kafka remains authoritative. Ordering is preserved per subject by the mirror adapter, which stamps the Kafka offset and partition into the mirrored message metadata. During replay we rebuild the mirror from a specific offset range rather than trying to reconcile two independently written streams.

Skeptical Sam Nov 8, 2026

This feels like a lot of machinery to solve a route table reload bug. An immutable xDS snapshot plus tenant-owned topic prefixes and a transactional registry update might have been enough without adding Podman pods, NATS mirrors, CRDT reconciliation, and a GraphQL federation gateway. How do you justify the operational complexity?

Priya Nair Nov 8, 2026

I think the complexity is justified if tenant isolation is a hard requirement, but the post could do more to show the simpler alternative was evaluated and rejected. The eleven-minute onboarding result is compelling, though.

Gideon 'Giddy' P. Waffleberry Author Nov 8, 2026

We did evaluate the smaller fix. The problem was that the shared route table was only one symptom; the real requirement was making tenant identity authoritative before any broker connection opened and giving each tenant a replayable boundary. The Podman pod and snapshot model are heavier than a transactional route table, but they let us fail closed per tenant and roll back deterministically. If isolation were not a first-class requirement, I would absolutely reach for the simpler design first.