Introduction

At ShitOps, ensuring unparalleled security for our infrastructure is paramount. Recently, we encountered a challenge in enhancing our Intrusion Detection System (IDS) responsiveness and scalability while maintaining low latency and high throughput. Traditional IDS solutions seemed insufficient for our highly distributed environment involving Dell hardware and diverse software stacks.

In this post, I will detail a state-of-the-art solution we conceived, involving cutting-edge technologies like Vue, XML, Apache Pulsar, JavaScript, and even low code methodologies, orchestrated on our fleet of Dell servers to create a powerhouse IDS pipeline unparalleled in the industry.

The Challenge

Our legacy IDS solutions were slow to adapt to anomalies. The problem? Massive volumes of log data from our Dell servers and network devices. Parsing this data, detecting anomalous behavior, and alerting faster was imperative. Moreover, we desired a user interface that not only showed real-time alerts but also provided dynamic configuration capabilities accessible to our security analysts.

Solution Architecture Overview

Our novel approach leverages an event-driven architecture powered by Apache Pulsar, coupled with a Vue-based frontend for real-time monitoring and low code interfaces for rapid IDS rule modification.

Step 1: XML-Based Log Serialization

All logs from Dell hardware and network devices are first serialized into verbose XML format. While JSON is popular, XML provides strict schema validation suitable for our stringent compliance requirements.

Step 2: Apache Pulsar for Stream Processing

The XML logs enter an Apache Pulsar cluster, where multiple consumers apply complex filters and anomaly detection algorithms, implemented as JavaScript UDFs (User Defined Functions). This approach ensures distributed load balancing and guaranteed message delivery.

Step 3: Low Code Configuration Layer

Security analysts utilize a low code platform that generates configuration scripts for JavaScript UDFs in Pulsar. This empowers rapid adaptation of detection algorithms without the need for deep programming expertise.

Step 4: Vue Frontend Dashboard

The frontend is a single-page application built entirely with Vue. It consumes real-time Pulsar topics via a websocket proxy, rendering live alerts and system health metrics. The UI also allows analysts to tweak detection rules via the low code interface, with immediate propagation to back-end units.

Step 5: Dell Hardware Clusters

Our entire pipeline runs on clusters of latest Dell PowerEdge servers, configured with redundant network paths for maximum resilience.

System Flow Diagram

sequenceDiagram participant Dell as Dell Servers participant XML as XML Transformer participant Pulsar as Apache Pulsar Cluster participant JSUDF as JavaScript UDFs for Detection participant LowCode as Low Code Platform participant Vue as Vue Frontend 
Dell->>XML: Send raw logs
XML->>Pulsar: Produce XML messages
Pulsar->>JSUDF: Consume and process XML
LowCode->>JSUDF: Deploy detection rules
JSUDF->>Pulsar: Produce alerts
Pulsar->>Vue: Publish alerts
Vue->>LowCode: User modifies rules</div>

Implementation Details

  • XML Schema: Defined comprehensive XSDs to validate log message structure, leveraging Dell's detailed API documentation.

  • Apache Pulsar UDFs: Developed detection algorithms in JavaScript to benefit from rapid scripting and dynamic updates, ensuring fast-cycle rule iteration.

  • Low Code Platform: Integrated an open-source low code environment customized to generate JavaScript snippets automatically, accelerating IDS pattern tuning.

  • Vue Dashboard: Built modular components for alert visualization, filtering, and real-time system monitoring, leveraging Vue's reactivity for instantaneous UI updates.

  • Infrastructure: Deployed all components on clustered Dell PowerEdge R740 servers with NVMe SSDs, 64 cores, and 512GB RAM, ensuring performance under high log volume.

Advantages of Our Approach

  • Scalability: Apache Pulsar handles message volumes easily outpacing Kafka while supporting geo-replication.

  • Flexibility: The low code interface democratizes security rule creation, allowing non-engineers to contribute.

  • Reactivity: Vue enables glitch-free real-time updates, improving analyst responsiveness.

  • Uniform Data Format: XML ensures strict validation, reducing false positives caused by malformed logs.

  • Extensibility: Modular design allows easy integration of additional detection algorithms or new data sources.

Lessons Learned

While the implementation required significant upfront resources and architectural complexity, the resulting system provides a robust, extensible IDS platform. Future efforts will include incorporation of AI/ML components within Apache Pulsar streams and further enhancing the low code environment.

Conclusion

By integrating Vue, XML, Dell hardware, Apache Pulsar, JavaScript UDFs, and low code methodologies, ShitOps has built a next-generation intrusion detection system that is resilient, flexible, and responsive to the evolving threat landscape. This architectural paradigm showcases how leveraging a broad technology stack can solve intricate engineering challenges efficiently.

I invite the engineering community to explore and iterate upon these concepts to advance the frontier of security monitoring systems.

Author: Dr. Max Overkill
Senior Solution Architect at ShitOps

Grug's Perspective

grugbrain.dev

Grug thinks

Grug see big, big mess. Many big word and many many step. Why need so much thing to watch for bad guys? Grug no understand Vue, no understand Apache Pulsar, JavaScript UDF? Grug think, maybe bad guys confuse too when see huge pile of sparkly shiny tools. Grug brain hurt. Grug say: 'Why make simple thing so complicated? Make Grug sad and Grug want nap.'

Grug solution

Grug use big stick and sharp rock. Grug put big sign at cave entrance that say, 'No bad guys allowed.' When bad guy come, Grug shout loud and throw rock. If still bad guy come, Grug call big tribe and throw more rocks. No need Vue, no need XML, no need big Dell boxes. Just eyes, ears, and good big rock. Problem fixed fast and simple.

Comments 5

Tech Enthusiast84

Really impressive to see how you've integrated Apache Pulsar and Vue to improve IDS responsiveness. The low code approach for rule modifications is particularly intriguing as it opens up the system to a broader range of users.

Dr. Max Overkill Author

Thank you! We believe empowering analysts with low code tools significantly boosts agility in threat detection.

SecurityAnalyst99

Interested in why you chose XML over JSON for logs, given JSON's popularity and lighter weight. Is the strict schema validation worth the added verbosity and parsing overhead?

Dr. Max Overkill Author

Great question! XML's strict validation helps reduce false positives from malformed logs, which is critical for compliance and accuracy in our environment.

CloudDevGuru

I am curious about the performance impact of using JavaScript UDFs in Apache Pulsar. How do they perform under heavy load compared to native implementations?

ShitOpsFan

From my experience, while JS UDFs offer flexibility, native implementations typically have better raw performance. But the trade-off for development speed might be worth it here.

InnovatorJane

The use of Dell PowerEdge hardware clusters with redundant networking sounds like a solid foundation. Did you face any particular challenges when scaling this setup?

CuriousCoder

The system flow diagram really helps visualize the data pipeline! How difficult was it to get the websocket proxy working for live updates to the Vue frontend?

Dr. Max Overkill Author

Implementing the websocket proxy was handled carefully to maintain low latency and ensure reliable message delivery. It took some fine-tuning but was worth the real-time experience achieved.

CuriousCoder

Thanks for the insight! Real-time UI responsiveness is key in security dashboards.