Every Monday at 09:00, our network reliability team opens a Microsoft Power Point deck that summarizes switch health across 1,800 top-of-rack devices. The deck is the contract between the NOC, capacity planning, and the platform team. It tells us which switch drives are approaching saturation, which ports are flapping, and which firmware cohorts need attention. The deck is generated by a homegrown cron job that has served us well for years. Then came the firmware upgrade.
The Problem: A Switch Firmware Change Broke the Drive Story¶
After the 7.4.2 switch firmware rollout, the telemetry schema changed. Drive health counters arrived in a new format, the old parser silently dropped them, and the Monday deck started showing drive utilization as unknown. Alerts did not fire because the alert rule only looked at parsed values. By Wednesday, the team was making capacity decisions from stale slides. The AI Retraining loop that tunes our anomaly thresholds was also stale, because it had been trained on the old schema. We needed a durable path from switch telemetry to the deck and to alerts. We decided to build a dedicated control plane around Argo Workflows.
The Architecture: Events In, Slides Out¶
The pipeline treats every switch and every drive as a first-class signal. Collectors run as a DaemonSet on the edge Kubernetes clusters, scraping switch APIs, gNMI streams, and drive SMART data. They publish to Kafka. Argo Workflows owns the DAG from raw signal to rendered deck. ClickHouse stores time series. A vector database stores slide embeddings for the AI Retraining service. A Windows node pool runs the Microsoft Power Point rendering job through Office COM automation. A homegrown alert engine evaluates both the time-series data and the visual diff of the generated deck.
Ingesting Switch and Drive Signals¶
Each collector exposes a small gRPC service and a feature switch that lets us enable new parsers per firmware cohort. The switch is controlled through a custom resource, so a canary cohort can move first. For every switch we emit port counters, temperature, PSU state, and drive wear. For every drive we emit SMART attributes, rebuild progress, and throughput. The collector normalizes everything into a protobuf envelope with a schema version. Kafka Connect sinks the envelopes into ClickHouse and object storage. We keep raw payloads for 30 days so the AI Retraining job can replay history when a parser changes.
AI Retraining on Slide-Aware Telemetry¶
The AI Retraining service is not just retraining anomaly thresholds. It also learns which charts and tables in the Microsoft Power Point deck are likely to be misinterpreted by humans. We extract slide text, chart captions, and table headers, embed them, and compare them with the telemetry embeddings. When the visual narrative drifts from the underlying switch and drive data, the service flags the deck for a retrain. The retraining job runs in Argo Workflows on a schedule and on schema change events. It writes model versions to a registry and updates the homegrown alert engine with new thresholds. This keeps the alerts aligned with how the Monday deck actually tells the capacity story.
Rendering Microsoft Power Point Decks with Argo Workflows¶
The rendering step is where many pipelines stop. Ours does not. An Argo Workflows template schedules a Windows pod, mounts the model output and the ClickHouse extracts, and drives Microsoft Power Point through COM automation. The pod opens a master deck, populates charts, adds speaker notes with drive IDs, and exports a PDF for archival. The generated deck is written to SharePoint with an immutable name and a content hash. A downstream workflow compares the new deck with the previous one and publishes a slide-level diff. If the diff touches a drive that is above threshold, the homegrown alert engine opens an incident. The same Argo Workflows DAG also posts a summary to Slack and sends PagerDuty alerts for the most urgent switch cohorts.
Homegrown Alerts and the Weekly Switch Review¶
We kept the alert logic homegrown because it needs to understand both telemetry and slide semantics. The engine evaluates rules against ClickHouse, checks the deck diff, and consults the latest AI Retraining model. A rule can say: if a switch drive has less than 15 percent headroom and the deck marks it green, fire an alert. Another rule can say: if the firmware cohort changes and the deck still shows the old schema, fire an alert. The alerts include the switch hostname, drive serial, deck hash, and the exact slide number. During the Monday review, the NOC opens the deck, clicks through the annotated slides, and sees the same incident IDs that were sent to PagerDuty. The switch and drives are no longer a blind spot.
Rollout and Results¶
We rolled out the pipeline in three waves. Wave one covered 40 canary switches with the feature switch enabled. Wave two covered 400 switches in the latency-sensitive regions. Wave three covered the remaining fleet. Argo Workflows retries failed render pods automatically, and the homegrown alert engine suppresses duplicates by deck hash. We now generate the Microsoft Power Point deck in under 11 minutes for 1,800 switches and 7,200 drives. The AI Retraining job runs nightly and on every schema change. Alerts that used to arrive after the Monday meeting now arrive before the deck is published. The switch firmware team gets a slide-level diff instead of a spreadsheet investigation.
Lessons Learned¶
The biggest lesson is that telemetry is only useful when it reaches the artifact people actually read. For us that artifact is a Microsoft Power Point deck, so we made the deck a first-class output of Argo Workflows. The second lesson is that AI Retraining must be tied to schema versions and to the visual narrative, not just to raw metrics. The third lesson is that a homegrown alert engine can be precise when it has the right context: switch identity, drive identity, deck hash, and model version. We are now extending the same pattern to power supplies and optics, with the same Argo Workflows DAG and the same alert contract. The switch drives are healthy, the deck is accurate, and Monday starts with facts instead of folklore.
Comments 5
Priya N.
I admire the ambition, but this reads like a Rube Goldberg machine: Kafka, Argo, ClickHouse, a vector DB, Office COM automation, and a homegrown alert engine that understands slide semantics. Why is a PowerPoint deck the contract instead of a dashboard or a versioned report? And isn't COM automation on Windows pods notoriously brittle?
Marcus L.
The COM automation part is what worries me. We tried driving Excel/PowerPoint from scheduled jobs and Office updates broke us at least once a quarter. Did you pin Office versions in the Windows node image, or are render pods allowed to drift?
Dexter Q. Sprocket Author
Fair pushback. The deck is the contract because the NOC, capacity planning, and platform team already make Monday decisions from it; replacing that social artifact first would have been a bigger migration. We pin the Windows node image and Office build, run render pods with retries, and publish the deck hash. The COM layer is ugly, but it is isolated in one Argo template.
Priya N.
That helps, but I still wonder whether the vector-DB slide-embedding loop is worth the operational surface area. Do you have evidence it catches real misinterpretations, or is it mostly a clever guardrail?
Aisha K.
The 'AI Retraining on slide-aware telemetry' section is the most interesting and the most suspicious. How do you actually evaluate whether a chart is likely to be misinterpreted by humans? It sounds like you're training a model to predict human confusion from slide text and telemetry embeddings. What's the ground truth?
Dexter Q. Sprocket Author
We do not claim to predict confusion perfectly. The service flags drift between the slide narrative and telemetry distributions, and we use NOC review notes plus incident outcomes as weak labels. It is a guardrail, not an oracle. If the model version changes, the alert engine records the version so we can audit why a rule fired.
Ben O.
Homegrown alerts that parse slide diffs are exactly the kind of thing that starts as a clever hack and becomes a 24/7 maintenance tax. The rule 'drive has less than 15 percent headroom and deck marks it green' should probably fail the pipeline, not page someone. Why keep that logic homegrown instead of pushing it into the data quality checks?
Rina S.
I partly agree, but the deck is what people read. If the deck says green while the drive is red, that is an operational incident, not just a failed build. The alert context with deck hash and slide number is actually useful.
Dexter Q. Sprocket Author
We do both. The pipeline fails on schema and render errors, but the semantic mismatch alert is intentionally separate because the deck can be technically valid and still tell the wrong story. The homegrown engine is small and rule-based; the hard part is keeping the context contract stable.
Tom H.
Under 11 minutes for 1,800 switches and 7,200 drives is a great result. What is the actual bottleneck: ClickHouse extracts, COM rendering, PDF export, or SharePoint upload? And when a render pod fails halfway through, does Argo retry the whole deck or just the slide batch?
Dexter Q. Sprocket Author
Rendering dominates, especially chart population and PDF export. We batch slides by switch cohort so retries are scoped, then recombine and hash the final deck. SharePoint upload is fast by comparison. The 11 minutes is the happy path; a bad render batch can add a few minutes.
Sam R.
This is a genuinely useful post, but the title promises 'keeping switch drives healthy' and the solution is mostly observability and reporting. What actually remediates a drive that is approaching saturation? Do alerts trigger replacement workflows, or is this still a human-driven capacity process?