Cloud-Native Event Sourcing for Stateless Camera Spam Detection using MariaDB and Python

Introduction¶

In the era of ubiquitous cameras and the increasing prevalence of spam images generated by malicious AirPods users (yes, Marvel fans!), our company ShitOps has encountered a challenging problem: how to efficiently detect and backup spam images captured by stateless cameras in a cloud-native environment, while handling massive concurrency, and maintaining an event-sourced audit trail.

The solution must leverage the power of MariaDB, Python, Git, and advanced event sourcing techniques. This blog post unveils our state-of-the-art architecture that addresses this complex challenge with a marvel of engineering.

Problem Definition¶

Our stateless cameras stream continuous images to our backend system. Some images are legitimate, while others constitute spam—irrelevant or malicious content sent repeatedly, consuming resources and polluting storage.

Traditional spam filtering doesn’t scale well with stateless devices and high concurrency. Moreover, regulatory compliance requires us to maintain an immutable audit trail of all detections and actions performed, including backups for recovery.

Hence, the problem boils down to:

• Detect spam images from stateless camera streams in real time.

• Handle concurrency at massive scale.

• Maintain an immutable log of all state changes in our system.

• Use cloud-native infrastructure for scalability.

• Incorporate efficient backup and recovery.

• Utilize familiar technologies like MariaDB, Python, and Git.

Architectural Overview¶

To solve this, we devised a multi-layered architecture:

1. Cloud-Native Stateless Microservices: Each microservice is containerized and orchestrated with Kubernetes, ensuring statelessness and scalability.

2. Event Sourcing with Git and MariaDB: We use Git repositories as the event store to track every event. Each image processed generates a commit, enabling complete traceability.

3. Python-Based Event Processors: These listen to Git events, process images, and trigger spam detection algorithms concurrently.

4. Concurrency Model: Utilizing Python’s asyncio with Celery distributed task queues for concurrent processing.

5. Spam Detection Core: Employing a custom machine learning algorithm trained on Marvel-themed spam images.

6. Backup Service: Periodic snapshot backups of Git repos stored in MariaDB blobs for disaster recovery.

7. Real-Time Dashboard: A web interface for monitoring spam detection metrics.

Detailed Solution Components¶

Event Sourcing with Git + MariaDB¶

Every image received triggers a commit into a dedicated Git repository. The commit message contains metadata about the image (timestamp, camera ID, etc.).

MariaDB stores the compressed Git repositories as binary large objects (BLOBs), serving as a highly durable backup mechanism.

This hybrid approach exploits Git’s excellent version control and MariaDB’s reliable storage.

Python Event Processor Workflow¶

The core processing service is written in Python, utilizing asyncio to handle thousands of cameras concurrently. Upon receiving a Git commit event (using webhooks), it:

• Clones the latest Git repo state.

• Extracts and processes the new image data.

• Applies the spam detection ML algorithm.

• Records results back as new commits.

• Triggers backup jobs if thresholds exceeded.

Spam Detection Algorithm¶

This marvel of ML ingenuity uses convolutional neural networks trained specifically on Marvel-themed spam images sent from AirPods devices.

The model detects spam with a high precision, ensuring low false positives.

Backup and Recovery¶

Backup jobs archive Git repo snapshots periodically into MariaDB BLOBs. This provides layered durability and rapid recovery path in case of failures.

System Flow¶

Concurrency Handling¶

Our Python async workers coupled with Celery tasks ensure high concurrency handling. This suits the stateless nature of the cameras well, where each image is independently processed without shared state.

Closing Remarks¶

This full-stack solution elegantly combines cloud-native microservices, event sourcing via Git, MariaDB to store backups, and Python-powered concurrency to tackle the sophisticated problem of spam detection from stateless cameras.

The synergy between these components enables a system that is scalable, auditable, and robust, keeping ShitOps at the forefront of engineering excellence.

Stay tuned for upcoming blog posts where we'll deep dive into the machine learning model specifics and the Kubernetes deployment strategies!

Until then, keep engineering marvels!