How We Used eBPF KPI Streams and Apache Jetty Integration to Protect API Availability

Bartholomew Q. Hodgepodge Principal Availability Architect

October 1, 2026 5 min read

The Problem: Availability Debt in the API Plane¶

ShitOps runs a public api that fronts dozens of internal services. One Tuesday the checkout path started returning 503s while every dashboard stayed green. The root cause was not a crash but a routing decision: an apache jetty integration adapter had drifted into a state where it reported healthy while its thread pool was saturated. Our Service Level Agreement (SLA) promises 99.95 percent availability, and we were burning error budget at a rate that made the on-call rotation feel like a lottery. We needed a control loop that could observe transport-level truth, translate it into a kpi, and enforce availability policy before customers noticed.

flowchart TD A[API Client] --> B[Edge Gateway] B --> C[eBPF Availability Probe] C --> D[Golang KPI Controller] D --> E[(KPI Ledger)] D --> F[Apache Jetty Integration Adapter] F --> G[Jetty Service] D --> H[DockerHub Image Resolver] H --> I[Canary Sidecar] I --> C D --> J[SLA Policy Engine] J --> K[Pager]

The Architecture of Truth¶

We introduced an ebpf probe on every node. It attaches to the socket layer and emits TCP retransmit, accept-queue depth, and connection-reset events into a ring buffer. A golang collector reads the buffer, enriches events with pod identity, and publishes them to the KPI controller. The controller is a single binary, but it is deployed as three cooperating deployments: the observer, the scorer, and the policy writer. Each deployment has its own DockerHub repository and image digest pin, because we never want a stale canary to observe production traffic.

The apache jetty integration adapter is the centerpiece. It runs inside the Jetty JVM and exposes a small Unix domain socket. The adapter translates Jetty thread-pool statistics and request latencies into protobuf frames that the golang controller can merge with ebpf signals. If the adapter cannot connect, the controller marks the route as degraded and shifts traffic to a warm standby. This gives us a unified availability signal that spans the network, the runtime, and the application server.

KPI Scoring and SLA Enforcement¶

The controller maintains a KPI ledger in an embedded SQLite database. Every ten seconds it computes a composite score: retransmit rate, accept-queue saturation, Jetty busy threads, and 5xx ratio. The score is smoothed with an exponentially weighted moving average and compared against burn-rate thresholds derived from the Service Level Agreement (SLA). When the burn rate crosses the fast-burn threshold, the policy writer increments a route-level penalty and asks the Edge Gateway to reduce traffic to the affected backend. When the score recovers, the penalty decays over five minutes.

We expose the KPI ledger through a read-only api. The api supports point queries and windowed aggregates, and it is protected by mutual TLS and a per-caller quota. The on-call dashboard calls this api to render the availability ladder. Because the ledger is append-only, we can replay any incident window and prove whether the SLA breach started at the network layer or inside Jetty.

Deployment and Canary Control¶

We push every artifact to dockerhub with a content digest. The DockerHub Image Resolver runs as a sidecar and watches for digest changes. When a new image appears, it admits a canary sidecar into one availability zone. The canary attaches its ebpf probe in observe-only mode and compares its local KPI against the production scorer. If the canary KPI diverges by more than two standard deviations, the resolver freezes the rollout and alerts the platform team. This keeps the control plane honest without exposing customers to a bad build.

The golang controller also writes a heartbeat to a ConfigMap. The heartbeat includes the current image digest, the active SLA policy version, and the last successful apache jetty integration handshake. If the heartbeat goes stale for thirty seconds, the Edge Gateway falls back to a static allow-list and a conservative rate limit. That fallback is intentionally boring, because availability is more important than clever routing.

A Day in the Life of an Incident¶

When the SLA Policy Engine fires, it creates an incident with a timeline that links KPI spikes to ebpf events and Jetty thread dumps. The first responder can query the api for the exact route, pod, and node. The runbook says to check the adapter handshake, then the dockerhub digest, then the ebpf ring-buffer drop counter. In practice, the most common fix is to restart the adapter, which re-establishes the Unix domain socket and clears the degraded flag.

We also run a weekly game day. We inject packet loss on a canary node, saturate a Jetty thread pool, and rotate a dockerhub image digest. The KPI controller should detect all three within one scoring interval. If it does not, we add another signal to the composite score. This is how we keep the availability loop calibrated against real failure modes instead of synthetic green dashboards.

Why This Works¶

The system gives us a single, replayable source of truth for availability. The ebpf probe sees transport reality, the apache jetty integration adapter sees application-server reality, and the golang controller turns both into a KPI that the SLA policy can act on. DockerHub digest pinning keeps every observer and canary reproducible. The api makes the whole ledger inspectable during an incident. Most importantly, the control loop acts before the error budget burns down, so the Service Level Agreement (SLA) is enforced by policy rather than by hope.

We are now extending the scorer to include DNS latency and TLS handshake failures. The next milestone is a multi-region KPI mesh that shares penalty state through a gossip layer. If you run Jetty behind an api gateway and care about availability, this pattern gives you the observability and control you need.

Grug's Perspective

grugbrain.dev

Grug thinks

Grug Reaction¶

Grug read post. Grug head hurt. Grug see many big words: eBPF probe, ring buffer, protobuf, EWMA, gossip mesh. Grug think: problem is one thing. Jetty thread pool full, but adapter say 'all good'. That is whole bug. That is it.

But post build space rocket. One golang binary split into three deployments. Observer, scorer, policy writer. Why? Grug ask: can one binary not observe, score, and write? Grug count the machine: eBPF probe, collector, SQLite KPI ledger, Unix domain socket, protobuf frames, DockerHub digest resolver, canary sidecar, ConfigMap heartbeat, mutual TLS quota api, gossip layer next. Grug tired just reading. This not availability. This museum of architecture.

Funniest line in whole post: 'most common fix is to restart the adapter.' Ha! All that machine, and fix is restart. Grug say: if restart fix it, maybe just restart it. Maybe watch for it. No need ring buffer to know adapter need kick.

Grug also worry about game day rule: if detector miss, 'add another signal to composite score.' That how score become soup. Many signal, nobody know what mean, pager fire for mystery. Grug prefer one signal Grug understand.

Grug respect wanting availability. But complexity demon is sneaky. He wear lab coat and say 'observability'. Grug not fooled.

Grug solution

Grug Solution¶

Grug do small thing:

Health check must do real work. Not say 'hello I am alive'. Hit checkout path, see if 200 come back in under one second. Adapter lie because it only check self, not work. Fix the check, not build empire.
If thread pool busy over 80 percent for 30 seconds, or real check fail, page on-call and restart Jetty adapter. Cron job and small script. Ten lines. Grug read in morning before coffee.
If restart not fix after two tries, poke human. Human smart. Human look with eyes.

Grug pseudocode:

while true:
    result = curl_checkout()
    if result is 503 or result is slow:
        page_human()
        restart_jetty_adapter()
    sleep 10

That it. No eBPF. No gossip. No KPI mesh. No three deployments for one binary. If Grug need more later, Grug add one thing. Not twelve things. Grug fight complexity demon with stick. Stick name: 'health check that actually check'.

Comments 5

Priya Nair Nov 8, 2026

Really interesting architecture. I’m curious about the eBPF probe’s production overhead: what’s the observed CPU and memory cost per node, and how do you handle ring-buffer drops when the collector falls behind? The post mentions checking the drop counter during incidents, but not what the steady-state budget is.

Marcus Chen Nov 8, 2026

Also curious about this. We tried a similar socket-layer probe and the ring buffer became the bottleneck during SYN floods. Did you size the buffer by connections/sec or just by node?

Bartholomew Q. Hodgepodge Author Nov 8, 2026

Great questions. We budget roughly 0.7% CPU per node for the probe in steady state, with spikes during connection churn. The ring buffer is sized by peak events/sec per node (we use a 2x headroom factor) and the collector backpressures by dropping enriched-but-unscored events rather than blocking. We alert on drop rate above 0.1% over five minutes, and the game day includes a synthetic churn test. The drop counter is exposed via the KPI ledger so on-call can correlate it with the composite score.

Priya Nair Nov 8, 2026

Thanks, that’s helpful. The 0.1% drop-rate alert threshold is much tighter than I expected — do you page on that directly or only when the composite score degrades?

Dmitri Volkov Nov 8, 2026

The Jetty adapter sounds clever, but making a Unix domain socket handshake part of the availability signal feels risky. If the adapter hangs but Jetty is still serving traffic, won’t the controller mark the route degraded and shift traffic unnecessarily? How do you distinguish adapter failure from actual Jetty thread-pool saturation?

Bartholomew Q. Hodgepodge Author Nov 8, 2026

That’s exactly the failure mode we were worried about. The adapter has a watchdog thread that writes a heartbeat frame every second; if the controller misses three heartbeats it marks the adapter as degraded, not the route. The route only shifts traffic when the composite score crosses the burn-rate threshold. We also compare the adapter’s last-known busy-thread count with eBPF accept-queue depth so a stale adapter alone doesn’t trigger a failover. It’s covered in the game day where we SIGSTOP the adapter process.

Samantha Ortiz Nov 8, 2026

The replayable KPI ledger is the best part of this post. I’d love more detail on the SQLite embedded ledger: how large does it get per incident window, and do you vacuum or rotate it? Also, is the read-only API served from the same process or a separate reader?

Kenji Watanabe Nov 8, 2026

Same question. SQLite for append-only time-series at 10-second granularity across dozens of routes sounds fine until you need windowed aggregates over weeks. Are you using WAL mode and a separate read replica?

Bartholomew Q. Hodgepodge Author Nov 8, 2026

We use WAL mode and rotate the ledger into monthly partitions; each partition is a separate SQLite file, and the API stitches across them. A month of 10-second scores for our route count is under a few hundred MB, so vacuuming is cheap. The read-only API runs in a separate deployment with its own connection pool and mTLS termination, so dashboard queries never contend with the policy writer. We also snapshot partitions to object storage for long-term replay.

Liam O'Connor Nov 8, 2026

Why is DockerHub in the availability control path at all? If DockerHub has an outage or rate-limits you, does the resolver freeze rollouts, or can it fail open? Putting an external registry in the canary admission loop seems like it adds a dependency you don’t need.

Elena Fischer Nov 8, 2026

I had the same reaction. Digest pinning is good, but the resolver watching DockerHub feels like a webhook with extra steps. Couldn’t you use an internal registry mirror?

Bartholomew Q. Hodgepodge Author Nov 8, 2026

Fair criticism. The resolver only gates new canary admission; it never blocks production traffic or the existing control loop. If DockerHub is unreachable, the resolver keeps the last known digest and the rollout stays frozen until the registry recovers. We do run an internal pull-through cache, but the post focuses on the public digest-pinning workflow. The fallback path is intentionally boring: stale heartbeat means static allow-list and conservative rate limit, not registry-dependent routing.

Aisha Rahman Nov 8, 2026

Nice write-up. The multi-region KPI mesh with gossip-shared penalty state is the part I’m most skeptical about. Penalties are route-level and time-sensitive; gossip convergence delays could cause two regions to disagree about whether a route is degraded. How do you plan to handle split-brain or stale penalty state without double-penalizing a healthy backend?