Building a Near-Real-Time CIFS Log Pipeline with Container Orchestration on Amazon

Bartholomew Q. Stackworthy Principal Platform Synergy Architect

October 14, 2026 4 min read

The Problem: CIFS Logs Stranded on Legacy Storage¶

ShitOps runs its order-routing workers on Amazon EKS. Those workers still write audit and diagnostic logs to a CIFS share hosted on a legacy NAS in our co-located data center. The CIFS mount is reliable, familiar, and deeply integrated into our batch jobs. The problem is that the logs sit on that share until an engineer manually copies them into a ticket or a spreadsheet. Security wants retention and search in Amazon. Support wants near-real-time visibility. Finance wants predictable storage growth. Our current process is a runbook, a cron job, and hope.

We evaluated the obvious options and rejected them. We needed container orchestration to own the lifecycle, CIFS access to remain first-class, Amazon-native logging to be the destination, and a pipeline that could tolerate NAS maintenance windows without losing a line. The result is the CIFS Log Ingestion Mesh.

Solution Overview: CIFS Log Ingestion Mesh¶

The mesh runs on Amazon EKS and treats the CIFS share as a source of truth, not as a destination. A CIFS CSI driver mounts the share into dedicated namespaces. A Fluent Bit DaemonSet harvests files, a sidecar validates file handles, Kafka absorbs bursts, Kinesis bridges into Amazon, Lambda enriches records, Step Functions routes them, and S3, OpenSearch, DynamoDB, and CloudWatch Logs provide retention, search, offsets, and operational logging.

flowchart TD A[CIFS NAS] --> B[CIFS CSI Driver on EKS] B --> C[Fluent Bit DaemonSet] C --> D[Kafka on EKS] D --> E[Kinesis Data Streams] E --> F[Lambda Enricher] F --> G[Step Functions Router] G --> H[S3 Archive] G --> I[OpenSearch Cluster] G --> J[DynamoDB Offsets] G --> K[CloudWatch Logs] L[EventBridge Scheduler] --> M[EKS Reconciliation CronJob] M --> B I --> N[Amazon Managed Grafana] K --> N

Step 1: Provision the CIFS Access Layer¶

We provision the EKS cluster with Terraform and install the CIFS CSI driver through Helm. Each log-producing namespace gets a PersistentVolumeClaim backed by a StorageClass that points at the NAS. The mount options are pinned: vers=3.0, uid=1000, gid=1000, file_mode=0660, dir_mode=0770, and noatime. An IAM role for service accounts grants the driver permission to read Amazon Secrets Manager, where the CIFS credentials live. A small sidecar called cifs-log-sentinel opens each file with a shared read lock, checks the file handle, and writes a heartbeat to DynamoDB. If the NAS drops a connection, the sentinel marks the volume as stale and the reconciliation CronJob remounts it.

Step 2: Harvest with Fluent Bit and Sidecars¶

The Fluent Bit DaemonSet runs on every node and tails the mounted CIFS path. We use the tail plugin with a multiline parser for Java stack traces, a Kubernetes filter for pod metadata, and a Lua filter for redaction. The sidecar handles CIFS-specific edge cases: files that are rotated on the NAS, files that are still being written, and files with Windows line endings. Fluent Bit forwards records to Kafka with a topic per namespace and a partition key based on the file path. This gives us ordering per file and parallelism across files.

Step 3: Container Orchestration with Kafka and Kinesis¶

Kafka runs on EKS through the Strimzi operator with three brokers spread across three availability zones. Topics are created with a replication factor of three and a retention of seven days. A Kinesis bridge consumes from Kafka and writes to an Amazon Kinesis Data Stream. This bridge is the only component that knows about Amazon. If Kinesis throttles, Kafka holds the backlog. If Kafka rolls, the bridge retries with exponential backoff. Container orchestration gives us rolling upgrades, pod disruption budgets, and node autoscaling without touching the CIFS share.

Step 4: Enrichment and Routing with Lambda and Step Functions¶

A Lambda function consumes from Kinesis, parses the record, adds tenant, environment, and retention class, and emits a normalized event. A Step Functions state machine routes the event. Audit records go to S3 with Object Lock. Search records go to OpenSearch. Offset records go to DynamoDB. Operational records go to CloudWatch Logs. The state machine has retry and catch blocks for every task, plus a dead-letter queue for poison records. EventBridge Scheduler triggers a reconciliation CronJob every fifteen minutes to compare DynamoDB offsets with the CIFS directory listing and replay any gaps.

Step 5: Observability and Retention¶

OpenSearch provides search and dashboards. Amazon Managed Grafana reads from OpenSearch and CloudWatch. S3 lifecycle policies move audit records to Glacier after ninety days. CloudWatch Logs holds pipeline health and error logs. DynamoDB tracks file offsets and sentinel heartbeats. We alert on lag, mount staleness, Lambda errors, and Step Functions failures. The result is a complete logging path from CIFS to Amazon, with container orchestration managing every moving part.

Why This Works¶

The CIFS share stays where it is. The logs become searchable, retainable, and auditable in Amazon. Each concern has its own component: access, harvest, buffering, enrichment, routing, storage, and observability. The mesh handles NAS maintenance windows, Kinesis throttling, and pod restarts without manual intervention. It is the cleanest way we have found to turn a legacy CIFS logging problem into a modern Amazon-native logging platform.

Grug's Perspective

grugbrain.dev

Grug thinks

Grug's Reaction¶

Grug read post. Grug need lie down on cave floor. Problem is logs on CIFS share. Post answer is: CSI driver, sentinel sidecar, Fluent Bit, Kafka, Kinesis, Lambda, Step Functions, S3, OpenSearch, DynamoDB, CloudWatch, EventBridge, Grafana, Terraform, Helm. That not pipeline. That whole village trying to carry one log file. Grug count moving parts. Grug ran out of fingers. Grug see Kafka on EKS with three brokers and Kinesis bridge and Lambda and state machine. Each part can fail. Each failure need own dashboard. Grug say: if NAS hiccup, now you debug CIFS driver plus Kafka lag plus Kinesis throttle plus Step Functions retry plus DynamoDB offset plus sentinel heartbeat. That not resilience. That is failure-themed escape room. Grug also notice sentinel opens file, checks file handle, writes heartbeat. Grug ask: why file need heartbeat? File is not pet. Grug suspect team wanted use every AWS service before deadline. Grug respect wanting search and retention. But Grug not respect calling a Rube Goldberg machine a mesh. Grug say: complexity very expensive. Simplicity cheaper and sleep better.

Grug solution

Grug's Simple Solution¶

Take one small Linux box. Could be EC2. Could be old server under desk if boss allow.
Mount CIFS share. Put credentials in file. Done.
Write tiny script: rsync -a --remove-source-files or smbclient copy new log files to local temp. Then aws s3 sync temp folder to S3 bucket. Or if no rsync, use one loop with cp and aws s3 cp. Grug not fussy.
Run script from cron every minute. Near-real-time enough for logs. If security want faster, every ten seconds. Cron not care.
S3 lifecycle rule move old logs to Glacier. S3 keep retention. Athena query S3 when support want search. Or grep before delete. Grug hear grep is search engine for adults.
CloudWatch alarm if script exit code not zero. One alarm. One dashboard. One thing to fix.

No Kafka. No Kinesis. No Lambda. No Step Functions. No OpenSearch cluster. No DynamoDB offset table. No sentinel sidecar. No mesh. One script, one bucket, one cron. If NAS down, script retry next minute. If box down, restart box. If logs pile up, script catch up. Grug call that pipeline. Grug call other thing expensive panic attack.

Comments 8

Skeptical SRE Nov 8, 2026

This is a genuinely interesting architecture, but I keep wondering whether a CIFS-to-S3/OpenSearch pipeline really needs Kafka, Kinesis, Lambda, Step Functions, DynamoDB, and a reconciliation CronJob all in the same path. It feels like the logging equivalent of building a space shuttle to cross the street. What failure mode forced the Kafka layer specifically?

Cost-Conscious Architect Nov 8, 2026

I had the same reaction. Kafka on EKS plus Kinesis plus Step Functions is a lot of fixed cost for what sounds like batch audit logs. Did you compare this against a simpler Fluent Bit to Firehose path?

Bartholomew Q. Stackworthy Author Nov 8, 2026

Fair challenge. Kafka exists because the CIFS share can produce bursts when batch jobs rotate large files, and we needed a buffer that survives Kinesis throttling without backpressuring the Fluent Bit DaemonSet. Kinesis is the Amazon boundary, not the primary buffer. We did evaluate Firehose, but it gave us less control over per-file ordering and replay semantics. The reconciliation loop is specifically there because we do not trust a single delivery path.

Kafka Fan Nov 8, 2026

The Kafka-first design makes sense to me. CIFS mounts are flaky enough that I would not want the Amazon write path directly coupled to the harvester. The real question is whether you actually need Kinesis after Kafka, or whether you could use MSK Connect or a Lambda consumer group.

CIFS Veteran Nov 8, 2026

The cifs-log-sentinel sidecar is the part I want to hear more about. Shared read locks on CIFS can behave very differently across NAS vendors, and file handle validation is not always portable. What happens when a file is rotated while Fluent Bit still has an open handle? Do you get duplicates, gaps, or both?

Bartholomew Q. Stackworthy Author Nov 8, 2026

We see both if we are careless, which is why the sentinel writes a heartbeat and the reconciliation job compares DynamoDB offsets against the directory listing every fifteen minutes. Fluent Bit tracks the file by path plus inode where possible, and the sidecar signals when a handle is stale. We still accept at-least-once delivery and deduplicate downstream on record hash plus file path plus offset.

Security Sam Nov 8, 2026

Storing CIFS credentials in Secrets Manager and granting access through IRSA is the right call, but I would like more detail on rotation and least privilege. Does the CSI driver get read-only access to the entire share, or is it scoped per namespace?

Bartholomew Q. Stackworthy Author Nov 8, 2026

The CSI driver role can read only the specific secret ARN, and the mount is scoped by the StorageClass and PVC to the namespace path. We rotate credentials with a Lambda job that updates Secrets Manager, then the reconciliation CronJob remounts affected volumes during a maintenance window. It is not seamless yet, but it is auditable.

Security Sam Nov 8, 2026

That is better than most implementations I have seen. The remount dependency is worth calling out more prominently because it means credential rotation is not fully zero-downtime.

Platform Engineer Nov 8, 2026

Nice write-up and the mermaid diagram is actually useful. The part I am stealing is the separation between the CIFS access layer and the harvest layer. Too many teams try to make Fluent Bit own the mount lifecycle and then wonder why node restarts cause missing logs.

Observability Lead Nov 8, 2026

I like the pipeline, but the alerting section worries me. You are alerting on lag, mount staleness, Lambda errors, and Step Functions failures across Kafka, Kinesis, Lambda, Step Functions, OpenSearch, DynamoDB, and CloudWatch. How do you avoid alert fatigue when the CIFS NAS has a maintenance window?

Bartholomew Q. Stackworthy Author Nov 8, 2026

We suppress mount-staleness alerts during declared NAS maintenance windows and page only if the sentinel heartbeat is missing outside those windows. Lag alerts are tiered by severity, and the reconciliation job emits a single health metric so we do not alert on every downstream symptom. It took a few iterations to get there.

On-Call Human Nov 8, 2026

Tiered lag alerts are essential here. I would also add a runbook link directly in the alert payload, because debugging a Step Functions catch block at 3 a.m. is not fun.

Data Reliability Engineer Nov 8, 2026

What does the replay path actually guarantee? The Step Functions router plus DynamoDB offsets plus S3 Object Lock sounds robust, but at-least-once delivery means duplicates are inevitable. Are you doing idempotency keys in the Lambda enricher, or relying on OpenSearch/S3 deduplication later?

Bartholomew Q. Stackworthy Author Nov 8, 2026

The Lambda enricher emits a deterministic event ID from file path, offset, and record hash. S3 uses that as part of the object key, and OpenSearch documents use it as the document ID. That gives us idempotent writes even when the replay path duplicates records.

Cloud Economist Nov 8, 2026

I appreciate the honesty about the legacy NAS, but I would love a cost breakdown. Kafka on EKS with three brokers across three AZs, Kinesis, Step Functions, Lambda, OpenSearch, DynamoDB, and S3 Object Lock is a lot of line items. Is this cheaper than just lifting the logs into S3 with a managed connector?

Junior DevOps Nov 8, 2026

This is a great end-to-end example. One small question: why EventBridge Scheduler instead of a Kubernetes CronJob for the reconciliation job? You already have EKS, so it seems like one more control plane to manage.

Bartholomew Q. Stackworthy Author Nov 8, 2026

We use EventBridge Scheduler because the reconciliation job needs to survive cluster upgrades and node drain events without being tied to a particular namespace or node pool. It also gives us a clean audit trail of invocations in CloudWatch. A Kubernetes CronJob would work, but we wanted the trigger outside the cluster it is reconciling.