The three-second ticket enrichment tax

At ShitOps, support agents open a ticket and wait about three seconds while the console fetches account tier, product telemetry, SLA state, and recent incidents. The data lives in the microsoft Graph change feed, a billing service, and a product analytics store. The old nightly Extract, transform, and load (ETL) job copied everything into a reporting table. By the time an agent sees the ticket, the context is stale and the coffee is cold. We decided the only sane answer was a regional, event-driven ETL mesh.

flowchart LR A[Microsoft Graph Webhook] --> B[QUIC Edge Ingress] B --> C[Beam Extract DoFn] C --> D[SSD-backed Transform Cache] D --> E[Beam Load DoFn] E --> F[Support Console] D --> G[Continuous Development Canary] G --> C

Architecture: every event gets a fast lane

We route every webhook through the quic edge ingress. The edge terminates quic connections with 0-RTT where possible and forwards envelopes to Apache Beam on Kubernetes. A custom SplittableDoFn extracts the Graph payload, fans out to the billing API, and emits normalized records. Transform stages run against a tiered cache backed by local solid-state-drives attached through a CSI driver. Load stages perform idempotent upserts into Microsoft Dataverse and the support read model. Continuous development is wired into every stage: a commit opens a canary pipeline, replays sampled traffic, and promotes only if latency and correctness budgets pass.

The pipeline in detail

  1. Ingest: a regional QUIC ingress receives signed Graph notifications and writes them to a partitioned event log with tenant-aware ordering keys.

  2. Extract: Apache Beam reads the log, expands each envelope into enrichment tasks, and uses stateful timers to debounce duplicate ticket updates.

  3. Transform: workers resolve entitlement tier, SLA clock, open incidents, and product usage from a cache warmed by solid-state-drives. Every transform is versioned and shadow-tested.

  4. Load: idempotent writers upsert into Dataverse, the search index, and a Redis read model. A two-phase commit marker prevents partially enriched tickets.

  5. Feedback: OpenTelemetry spans flow from the edge to the console, so we can trace one ticket across every DoFn and cache shard.

Why Apache Beam is the right runtime

Apache Beam gives us one programming model across batch backfills and streaming enrichment. We can run the same pipeline on a local runner for developer laptops, a regional runner for production, and a replay runner for incident investigations. Windowing lets us separate real-time ticket updates from hourly backfills. State and timers make duplicate suppression a first-class concern instead of a pile of Redis scripts. The portable model also means we can swap runners as our platform strategy evolves, which is exactly the kind of optionality a support platform needs.

QUIC at the edge

The quic ingress is not decorative. Support traffic arrives from laptops, phones, and flaky office networks. QUIC keeps independent streams from blocking each other and gives us faster reconnects when agents switch networks. We terminate quic at the nearest regional edge, authenticate with mTLS, and forward an internal envelope to Beam over the cluster network. The edge also applies per-tenant rate limits, so one noisy integration cannot push ticket enrichment behind a backlog of billing events.

Continuous development as a first-class runtime feature

Every pull request builds a container, deploys a canary pipeline, and replays a sampled slice of production envelopes through the new code. The canary compares enrichment output against the current pipeline and blocks promotion if field-level drift exceeds our threshold. Feature flags let us move traffic by tenant, region, or agent cohort. Continuous development also covers configuration: cache TTLs, QUIC timeouts, and Beam parallelism are policy-controlled and audited. Nobody edits a production value by hand at 2 a.m. unless the runbook explicitly says so.

Reliability and observability

The mesh runs active-active across two regions. Each pipeline writes to a regional SSD-backed cache and replicates finished enrichment records to the other region. Circuit breakers fall back to a last-known-good snapshot if Dataverse slows down. We track p50, p95, and p99 latency from webhook receipt to console render, plus cache hit ratio, Beam watermark lag, and QUIC handshake failure rate. Grafana dashboards and PagerDuty alerts are generated from the same service catalog that owns the pipeline, so on-call engineers know exactly which shard, runner, or cache volume is unhappy.

Rollout and outcome

We piloted the mesh with one support pod, then expanded by region. The first week showed p95 enrichment latency around 180 ms and a cache hit ratio above 97 percent. Agents now see ticket context before the page finishes rendering. The nightly ETL job remains only as a reconciliation lane for finance and compliance. With Apache Beam handling extraction and transformation, QUIC handling edge transport, solid-state-drives handling the hot cache, and Continuous development handling safe rollout, ticket enrichment is finally a streaming product instead of a batch tax.