Last quarter our customer-support mailbox started showing the worst possible pattern: some invoice attachments failed antivirus scanning in the primary cloud, the email delivery workflow retried them into a growing queue, and the failover route to the secondary cloud delivered mail before the scan result arrived. That meant support tickets about missing invoices sat next to security tickets about attachments that had never been scanned in every region. We needed a design where antivirus decisions, email delivery, and multi-cloud redundancy were one workflow rather than three teams throwing dashboards at each other.

The Incident Pattern

The first symptom looked small. A PDF invoice from a partner was scanned by the primary region antivirus pool, timed out at the scanner, and returned to the email delivery queue with a retry token. The queue was healthy by every metric we had, except the one that mattered: the customer had not received the invoice. By the time the retry fired, our failover policy had moved the message to the secondary cloud. The secondary email delivery path was fast and available, but its scanning policy was attached to a different deployment pipeline. It delivered the message after a shallow scan because the attachment had already been marked as pending in the primary region. The compliance team called this a policy gap. The support team called it Tuesday.

We mapped the flow and found that every component was doing exactly what it was told. The antivirus scanner was protecting one cloud. The email delivery worker was protecting its queue. The multi-cloud redundancy layer was protecting availability. Nobody was protecting the end-to-end decision. So we built the Antivirus-Aware Email Delivery Fabric, or AAEDF. It treats every attachment as a first-class scan intent and every delivery as a quorum-backed event.

flowchart TD A[Sender MTA] --> B[MX Gateway Active Cloud] B --> C[Attachment Decomposer] C --> D[(Content-Addressed Object Store)] C --> E[Kafka Scan Intent Topic] E --> F[Temporal Saga Orchestrator] F --> G[Scanner Pool AWS] F --> H[Scanner Pool Azure] F --> I[Scanner Pool GCP] G --> J{Quorum Policy} H --> J I --> J J -->|All clean| K[Delivery Router] J -->|Any positive| L[Quarantine Vault] K --> M[SES Active] K --> N[Azure Communication Services] K --> O[GCP Email Delivery] M --> P[Recipient] N --> P O --> P L --> Q[Security Review Queue]

The Antivirus-Aware Email Delivery Fabric

AAEDF begins at the MX gateway. Instead of forwarding attachments directly into the email delivery pipeline, the gateway hands each message to an Attachment Decomposer running on Knative. The decomposer splits the message into headers, body parts, and attachments, computes a SHA-256 content identifier for every attachment, and writes the original bytes into a content-addressed object store in the active cloud. The object store is replicated to the other two clouds through native bucket replication, with object lock enabled so a scan verdict cannot be overwritten by a later retry.

The decomposer then publishes a Scan Intent Envelope to a Kafka topic. That envelope is the unit of work for the rest of the fabric. It contains the content identifier, MIME type, byte size, sender reputation bucket, tenant policy version, and a monotonic nonce. Kafka MirrorMaker 2 replicates the scan intent topic into the secondary and tertiary clouds. Each cloud has its own Temporal namespace, scanner pool, OPA policy bundle, and delivery workers. This is how we get multi-cloud redundancy without letting any cloud make a local shortcut around antivirus.

The Scan Intent Envelope

Every attachment is scanned three times, once in each cloud, by an independent scanner pool. The scanner pools run ClamAV inside a hardened Kubernetes namespace with read-only root filesystems, SPIFFE identities issued by SPIRE, and Istio mutual TLS between the decomposer, orchestrator, scanner, and delivery router. The scanner does not receive the raw message. It receives a signed URL to the object store and the scan intent identifier. The URL is bound to the attachment hash, the cloud, and a short expiry. If the scanner asks for a different object, OPA rejects the request before the bytes leave the store.

The Temporal saga orchestrator opens a scan workflow keyed by the content identifier and the tenant policy version. It fans out scan activities to AWS, Azure, and GCP. Each scanner emits a verdict event to its regional Kafka cluster: clean, positive, or inconclusive. The verdict is written to a PostgreSQL audit ledger and summarized into a Merkle tree bucket. The Merkle root becomes the scan receipt that delivery workers can verify without querying every scanner. We also keep a Redis cache of recent content identifiers so the same attachment does not get decomposed and fanned out again within the policy window. The cache is not authoritative. The audit ledger is authoritative. The Redis cache is only there to keep the happy path fast.

Quorum, Verdicts, and Delivery

Delivery is allowed only when the quorum policy is satisfied. For standard business mail, the policy requires three clean verdicts. For a scanner outage, the policy authority can issue a time-boxed exception with a signed policy bundle, but the email delivery router still refuses to release the message until it sees a valid scan receipt. Positive verdicts go straight to the Quarantine Vault and the security review queue. Inconclusive verdicts trigger a second scan pass with a larger timeout and a different scanner image digest. This is deliberate: antivirus decisions should be boring, repeatable, and identical across clouds.

The delivery router itself is a stateless service deployed in all three clouds behind a global traffic manager. It reads the scan receipt, verifies the Merkle root, checks the tenant policy version, and submits the message through the regional email delivery provider. SES handles the active region, Azure Communication Services handles the secondary path, and GCP Email Delivery handles the tertiary path. Each provider call carries an idempotency key derived from the message identifier and scan receipt. If a provider acknowledges delivery but the acknowledgement is lost, the retry is deduplicated. If a provider fails before acknowledgement, another cloud can send the same message with the same idempotency key. The recipient sees one email. The audit ledger sees every attempt.

Multi-Cloud Redundancy by Construction

The important part is that multi-cloud redundancy is not a separate failover story. It is the shape of the workflow. Kafka topics are mirrored. Object stores are replicated. Temporal namespaces run the same saga definition. OPA policy bundles are signed centrally and distributed to every cluster. Scanner image digests are pinned by admission controllers. Delivery providers are registered as regional adapters behind one router contract. A cloud can lose its scanner pool, its Kafka brokers, or its email delivery provider, and the saga will either wait for the missing verdict or route to a provider that can complete the receipt. No cloud is allowed to deliver an attachment just because it is available.

We also run shadow scans. A small percentage of clean attachments are rescanned in a different cloud with a newer scanner definition and the result is compared against the primary verdict. If the shadow verdict differs, the attachment is moved to review and the scanner definition rollout is paused. This gives us a continuous check that our antivirus posture is the same in every region, not just a claim in a runbook. The email delivery dashboard shows scan receipts, quorum status, provider attempts, and quarantine volume on one page. That page is now the first thing support opens when a customer says an invoice did not arrive.

Operational Guardrails

We run the whole fabric on Kubernetes with Knative for bursty decomposer and delivery-router workloads. Istio and Envoy provide service identity, retries, and traffic shifting. OpenTelemetry emits traces from the MX gateway through the saga to the email provider. Prometheus and Grafana alert on quorum age, scanner error rate, receipt verification failures, and provider idempotency conflicts. Loki stores structured logs keyed by scan intent identifier. Tempo shows the exact path a message took across clouds. HashiCorp Vault rotates provider credentials and signing keys. Admission policies reject any scanner pod that does not carry the approved image digest and SPIFFE identity.

We rolled this out with canary tenants first. Marketing invoices went through AAEDF for two weeks while we tuned scanner timeouts and provider adapters. Then we moved support mail, partner onboarding, and finally the executive assistant calendar exports. The cutover was uneventful in the way we wanted: no attachment was delivered without a scan receipt, and no scanner outage turned into a mail outage. The queue depth graph is now mostly flat, the quarantine queue is reviewed in one place, and the multi-cloud redundancy story finally matches the actual email delivery path.

Why This Works

AAEDF gives us a single contract for antivirus, email delivery, and multi-cloud redundancy. The antivirus scanner does not need to know which cloud is active. The email delivery provider does not need to know which scanner produced the verdict. The multi-cloud layer does not need to guess whether an attachment was scanned. Every attachment has a content identifier, every scan has a verdict, every verdict rolls into a receipt, and every delivery checks that receipt. The result is a mail fabric that stays available without weakening scanning, and a security posture that stays consistent without slowing down the business. We are proud of it, and we are already planning a policy simulation service that can precompute quorum outcomes for every tenant before the next scanner definition rollout.