The Incident That Started It All

Last month a P1 incident slipped past our on-call rotation for eleven minutes. The IDS fired, infrastructure dashboards went red, and the alert landed in a noisy channel while the on-call engineer was sprinting to a meeting. We had smart watch telemetry from the team: heart-rate variability, stress spikes, sleep debt. We had hackernews chatter about a dependency outage. We had sftp exports from the legacy billing host. None of it was connected. So we built PulseTriage.

The PulseTriage Pipeline

PulseTriage treats every on-call signal as a first-class event. A smart watch companion app exports biometric windows every 30 seconds to an sftp drop on the legacy jump host. A cron job on the ingest gateway polls that drop, validates checksums, and publishes to a Kafka topic. In parallel, an IDS collector normalizes Suricata alerts into the same event envelope. A hackernews sentiment poller runs every 90 seconds and tags stories by keyword. All three streams meet at the LSTM incident forecaster, a two-layer sequence model that predicts whether an alert should page a human, a manager, or the incident channel. The output is written to Redis and Postgres, then fanned out through a webhook relay.

The Diagram

flowchart TD A[Smart watch companion] -->|sftp export every 30s| B[Ingest gateway] B --> C[Kafka event bus] C --> D[IDS bridge] C --> E[hackernews sentiment poller] D --> F[LSTM incident forecaster] E --> F F --> G[Complex Architecture orchestrator] G --> H[MetalLB dashboard service] G --> I[Infrastructure as code pipeline] I --> J[Kubernetes cluster]

Why sftp Still Belongs in the Plan

The legacy billing host only speaks sftp, and we refuse to special-case it. Instead, we made sftp a general-purpose ingestion adapter. Every producer that cannot speak Kafka gets an sftp drop, a checksum manifest, and a retention policy. The ingest gateway treats sftp files exactly like API events, so the LSTM sees one schema and the IDS bridge sees one envelope. This keeps the infrastructure uniform and lets us reuse the same Terraform module for every new data source. We also run a small sftp health checker that emits Prometheus metrics and alerts if file arrival lag exceeds two minutes.

The LSTM Forecaster

The LSTM consumes a 20-minute window of features: IDS severity counts, smart watch heart-rate variability, hackernews negative sentiment, deploy frequency, and sftp file arrival lag. We trained it on six months of incident timelines and synthetic stress scenarios. The model outputs a triage score from 0 to 100. Above 80, PulseTriage pages the on-call engineer and posts a summary to the incident channel. Above 95, it also rings a smart watch haptic alert. The model is served by a FastAPI pod with a sidecar that logs every inference for audit. We retrain weekly on a CronJob and promote models only after the Complex Architecture Review Board signs the lineage manifest. This gives us a repeatable loop and avoids manual judgement during incidents.

Infrastructure as Code, MetalLB, and the Event Mesh

The whole pipeline is deployed with infrastructure as code. Terraform creates the Kubernetes cluster, namespaces, Kafka topics, and sftp credentials. Helm charts install the LSTM scorer, the hackernews poller, the IDS bridge, and the event gateway. MetalLB provides stable load balancer IPs for the gateway and for the dashboard. We use a dedicated MetalLB address pool so the smart watch companion app can post to a predictable endpoint over the VPN. The infrastructure as code pipeline runs on every merge and includes a policy check from the Complex Architecture Review Board. The board does not block deployments; it records the graph and confirms every service has an owner, a runbook, and a rollback chart. The event mesh uses OpenTelemetry traces, so every alert can be replayed from Kafka.

IDS, HackerNews, and Smart Watch Signals

The IDS bridge is the security backbone. It consumes Suricata alerts, maps them to service names, and attaches network context to each incident candidate. The hackernews poller adds external context by scoring stories for outage language and dependency keywords. The smart watch signal adds human context by tracking stress and recovery. When all three agree, the LSTM score rises quickly. When only one signal fires, PulseTriage lowers the page priority and waits for a second window. This is how we keep the pager honest. The IDS team gets a daily digest, the hackernews poller gets a rate limit, and the smart watch app gets a clear privacy notice in the onboarding flow.

Dashboard and Rollout

The dashboard shows a live triage score, the top three contributing features, and a dependency graph. The on-call engineer can acknowledge from a smart watch complication, which sends a signed webhook through MetalLB to the gateway. The IDS bridge adds context from network alerts, and the hackernews poller adds external context. We also expose a read-only sftp status page for the legacy billing team. We rolled out to two teams. In the first week, PulseTriage reduced time-to-acknowledge by 37 percent and false pages by 22 percent. The LSTM flagged a memory leak before the infrastructure dashboards crossed threshold. The IDS integration caught a port scan that would have been lost in the noise. The smart watch haptic alert woke one engineer at 2:14 AM, and that engineer closed a P1 in nine minutes.

Security and Compliance

Every sftp drop is encrypted with separate keys stored in Vault, and every event is signed with mTLS before it enters Kafka. The IDS bridge redacts payloads before the LSTM sees them, and the smart watch companion app sends only aggregated biometric windows, never raw sensor streams. We rotate credentials on a 30-day schedule through the infrastructure as code pipeline. The hackernews poller respects robots.txt and caches responses for 15 minutes. All audit logs go to object storage with a 400-day retention policy. The Complex Architecture Review Board reviews the threat model quarterly. This is how we keep a broad signal mesh safe without slowing down incident response.

Operational Guardrails

We run three environments: dev, staging, and incident-replay. Each environment has its own MetalLB pool, its own sftp drop, and its own LSTM model version. The infrastructure as code pipeline promotes changes from dev to staging to production only after a canary analysis. The canary analysis compares triage scores against a held-out incident set and checks that the IDS bridge is still emitting traces. We also keep a manual override in the dashboard, because no model should own the pager alone. The override writes an audit event, opens a ticket, and notifies the Complex Architecture Review Board. We are now expanding to all on-call rotations and adding a second LSTM for capacity forecasting. The next milestone is to replace the sftp drop with an sftp drop that has better metrics.