High Throughput Intrusion Detection Using Function as a Service Profiler on CentOS with Threema Integration

Introduction¶

In today’s rapidly evolving threat landscape, ShitOps is always on the lookout for cutting-edge methods to enhance our security posture. One critical component is the Intrusion Detection System (IDS). Leveraging the power of Big Data, function as a service (FaaS), and state-of-the-art profiling, we have developed a groundbreaking high-throughput IDS solution fully integrated with the Threema secure messaging platform.

This blog post will delve into the architecture and design of this system deployed on CentOS Linux servers. Our focus was to address the challenge of detecting malicious activity at unprecedented speeds while seamlessly correlating events across datasets, providing real-time alerts via Threema, and profiling suspicious behaviors using a dedicated profiler microservice.

Problem Statement¶

Traditional IDS implementations often suffer from scalability and throughput limitations. Given the massive volumes of network traffic and system logs generated daily in our environment, it is paramount to implement an IDS capable of the following:

• High throughput processing to keep up with data inflow

• Complex algorithmic analysis for anomaly detection

• Integration with secure communication platforms for alerting

• Deployment on a stable and customizable Linux system (CentOS)

• Utilization of function as a service to scale out processing

• Profiling capabilities to study attacker behaviors deeply

System Architecture¶

Our solution consists of several interconnected components:

1. Data Ingestion Layer: Collects logs and network packets in real-time using specialized Linux kernel modules configured on CentOS.

2. Event Processing Cluster: A fleet of serverless function containers deployed as FaaS endpoints processes data at scale. Each function runs a custom anomaly detection algorithm that leverages advanced statistical models and machine learning techniques.

3. Profiler Microservice: This component continuously profiles entities flagged as suspicious, analyzing patterns and feeding back into the detection algorithms to improve accuracy.

4. Threema Alert Gateway: All detected threats are prioritized and sent as encrypted notifications to the security team through a customized Threema API.

5. Big Data Repository: All raw and processed data are stored in a distributed file system for retrospective analysis and auditing.

CentOS Linux Deployment¶

CentOS was chosen for its stability and extensive support for security modules. The kernel was custom-compiled with additional hooks for enhanced packet inspection. We optimized the Linux network stack to handle high throughput and integrated the IDS components tightly into the OS architecture.

Detailed Algorithm Workflow¶

The algorithm uses a multi-phase approach:

• Preprocessing Function: Extracts relevant features from incoming data streams.

• Anomaly Scoring Function: Applies machine learning models trained on massive datasets to assign a threat score.

• Correlation Function: Aggregates events using complex graph algorithms to detect coordinated attacks.

• Profiler Trigger Function: Upon surpassing specific thresholds, triggers the profiler microservice.

• Alert Function: Formats alert messages for Threema transmission.

All these functions are deployed as discrete FaaS units, allowing independent scaling and maintenance.

Mermaid Diagram of the Workflow¶

K->>FaaS: Send data stream
FaaS->>FaaS: Preprocessing
FaaS->>FaaS: Anomaly Scoring
FaaS->>FaaS: Correlation
FaaS-->>Profiler: Trigger profiling
Profiler->>DB: Store profile data
FaaS->>DB: Store processed events
FaaS->>Threema: Send alert</div>