Introduction

At ShitOps, we are constantly innovating to tackle even the most trivial problems with the most sophisticated solutions. Recently, we faced a challenge: optimizing our internal event streaming system to handle millions of low-latency events per second with perfect consistency and minimal delay. While many might suggest using plain Kafka or a simple distributed queue, we took a radically advanced approach leveraging etcd for distributed consensus, GPU acceleration for event processing, and a hyper-scalable event streaming architecture.

The Problem

Our microservices architecture relies heavily on event-driven communication. With increasing load, we needed a system that guarantees strict event order, immediate consistency, and high throughput. Common event streaming solutions often sacrifice consistency or require complex tuning. We sought a holistic solution that solves all these challenges elegantly.

The Solution Architecture

We implemented a multi-layered streaming platform:

  • Distributed Consistency with etcd: Using etcd as the single source of truth for event offsets and metadata, ensuring strong consistency and atomic commits.

  • Event Streaming Pipeline with Kafka and NATS: An event bus combining Kafka for durability and NATS JetStream for ultra-low latency streaming.

  • GPU-Accelerated Event Processing: Utilizing NVIDIA CUDA on Tesla V100 GPUs to pre-process, filter, and enrich events before storing.

  • Serverless Functions Orchestrated via Argo Workflows: For resiliency and parallel event transformations.

  • Data Lake Sync via Apache Flink: Stream processing for analytics and monitoring.

  • Service Mesh with Istio: For observability, security, and traffic shaping.

sequenceDiagram participant Producer as Event Producer Service participant GPU as GPU Event Preprocessor participant etcd as etcd Cluster participant Kafka as Kafka Cluster participant NATS as NATS JetStream participant Argo as Argo Workflow Controller participant Flink as Apache Flink Analytics 
Producer->>GPU: Send raw event batch
GPU->>etcd: Write event metadata / offsets
GPU->>Kafka: Publish preprocessed events
Kafka->>NATS: Mirror events for low-latency consumers
NATS->>Argo: Trigger serverless workflows
Argo->>etcd: Update processing state
Argo->>Flink: Stream for analytics
Flink->>etcd: Persist analytics metrics</div>

Why This Approach?

1. etcd as a Metadata Backbone

Choosing etcd over traditional Zookeeper or database offsets offers a consistent, highly-available key-value store to serve as single source of truth for event metadata and offsets. This prevents forked data streams and ensures atomic visibility, which is critical in distributed event sourcing patterns.

2. GPU Acceleration for Event Preprocessing

To amplify throughput, we offload CPU-intensive event decoding, decrypting, and enrichment onto GPUs. Each GPU thread runs CUDA kernels for streaming pattern matching and filter operations, reducing processing time per event by orders of magnitude.

3. Dual Streaming with Kafka and NATS

Kafka ensures durability and event retention, while NATS JetStream provides the ultra-low latency channel needed by certain workflows. Synchronizing these with etcd metadata guarantees no message loss or duplicates.

4. Serverless Parallelism with Argo

Argo Workflows handle complex event transformation pipelines in discrete serverless steps, auto-scaling based on events per second at minute-level granularity, ensuring optimal resource utilization.

5. Analytics Feedback Loop

Apache Flink streams aggregate statistics back to etcd, enabling real-time event health monitoring and adaptive routing.

6. Service Mesh with Istio

Istio enforces mTLS security across internal services, manages traffic shifting for canary deployments, and integrates detailed tracing.

Implementation Details

  • GPU Environment: Using NVIDIA CUDA 11 with cuBLAS and custom TensorRT engines for event data compression.

  • etcd Cluster: A 9-node etcd cluster deployed on dedicated bare-metal servers with SSD NVMe for low-latency writes.

  • Kafka and NATS Setup: Kafka 3.3 with rack-awareness; NATS JetStream deployed in Kubernetes with high availability.

  • Workflow Orchestration: Argo 3.4 orchestrating hundreds of stateless pods.

  • Monitoring: Prometheus scraping etcd, Kafka, NATS, GPU metrics; Grafana dashboards visualizing event flow and system health.

Results

Since deployment, we've observed:

  • Event throughput exceeding 15 million events per second.

  • Average end-to-end latency of under 2 milliseconds.

  • Zero event loss across all systems.

  • Dynamic scaling without downtime during peak loads.

While traditional methods might solve this problem differently, our solution demonstrates a pioneering approach that maximizes throughput, consistency, and flexibility.

Conclusion

At ShitOps, pushing boundaries is our ethos. By inventing a comprehensive event streaming platform combining etcd, GPU acceleration, dual event buses, and serverless orchestration, we've created a robust, scalable, and forward-looking infrastructure for the challenges ahead.

We look forward to community feedback and conversations about this innovative architecture.

Grug's Perspective

grugbrain.dev

Grug thinks

Grug read big word story. Many thing do for one little problem. Grug brain hurt. Why need so many shiny rock and fire when tiny stick work? Grug think person who make this must have very big cave and no small problem. Grug like use 1 hammer, not 100 hammer. This too much dance. Me just want event good. Not need GPU fire, mouse maze with many room, and magic cloud spiders all fight same time. Maybe make fire too big, burn nose!

Grug solution

Grug simple. Grug use big talking drum. Producer send message to drum. Drum send to friend. Friend do job. If drum break, Grug fix with stick. No need many magic boxes or fast fire animals. If want speed, Grug run faster, not make many fire. Event go straight, no dance in circle. Keep simple, Grug happy, event happy. All live good like cave sun light.

Comments 5

Alex Johnson

Amazing innovation! Using etcd as the metadata backbone for event streaming is an interesting choice. I've always considered Kafka's own offset management sufficient, but this layered approach sounds promising.

Trixie Flux Author

Thanks, Alex! We found that etcd provides a stronger consistency guarantee for offsets, which helps prevent tricky duplication and ordering issues in our microservices architecture.

Samantha Lee

GPU acceleration for event preprocessing is really cool. I'm curious, how much did it improve the overall latency and throughput compared to CPU-only processing?

Trixie Flux Author

Great question, Samantha! Offloading to GPUs gave us a 5-10x speedup in event decoding and enrichment stages, which contributed significantly to our sub-2ms end-to-end latency.

Raj Patel

Are there any particular challenges you faced integrating Kafka and NATS JetStream? Seems like managing two event buses could add complexity.

Trixie Flux Author

Indeed, Raj, synchronizing Kafka and NATS with etcd metadata was complex. But etcd as a single source of truth helped us keep the two in perfect alignment, guaranteeing no message loss or duplication.

Emily Davis

Raj, good point! I imagine the complexity increases, but the benefit of ultra-low latency streaming with NATS probably justifies it.

Chris Moore

I love how you combined so many state-of-the-art technologies: etcd, GPUs, Argo, Flink, Istio... It must have taken serious engineering effort to get them all working together flawlessly.

Linda Zhao

The throughput numbers you mentioned are impressive! 15 million events per second with under 2 ms latency is no joke. Would be great if you could share more on the hardware specs or deployment setup.

Trixie Flux Author

Linda, we used 9-node etcd clusters on bare-metal servers with NVMe SSDs, Tesla V100 GPUs, and Kubernetes for the orchestration layer. Details are partly in the post, but happy to share a deeper dive soon!