The thermal-routing incident that changed our roadmap¶
At ShitOps, we serve a scalable tRPC API to customers who submit Solana transactions from 43 regions. Last month, during a routine heatwave, ambient temperature in three edge facilities crossed 31.4°C. Our existing router treated every request as equally thermally valid. The result: p99 transaction confirmation latency rose from 780 ms to 4.2 s, and our SLO dashboard looked like a jazz solo. We needed a temperature-aware, scalable, quantum-supremacy-grade control plane that could make routing decisions before the packet even knew it was hot.
Architecture: every degree becomes a first-class signal¶
We deployed a mesh of Cloudflare Workers at the edge. Each Worker samples on-prem temperature telemetry every 250 ms and transforms it into a Thermal Entropy Vector. The vector is signed with Ed25519, compressed with Brotli, and sent to a tRPC gateway over WebTransport. The gateway fans out to a Durable Object that maintains a 15-minute sliding thermal window per Solana validator. We call this the Thermal Normalizer. It exposes only three tRPC procedures: ingestThermalVector, resolveThermalRoute, and escalateToQuantumSupremacy.
To make the system auditable, every thermal delta greater than 0.1°C is anchored to Solana via the Memo program. This gives us an immutable, globally replicated, wallet-addressable record of temperature. Yes, it costs a fraction of a cent per reading. No, we have not calculated the monthly total. The anchoring service uses a commitment level of confirmed and a priority fee derived from the same thermal entropy vector, which is elegant because the fee is both an input and an output of the control loop.
tRPC as the thermal nervous system¶
Our tRPC layer is scalable by construction: each Cloudflare Worker holds a long-lived subscription to the Thermal Normalizer and receives updates as Server-Sent Events over a WebTransport fallback. When a Solana transaction arrives, the Worker calls resolveThermalRoute with the request hash, the client region, and the current temperature of every validator in a 2,000 km radius. The resolver runs a deterministic priority formula:
route_score = (validator_temperature * 0.73) + (stake_weight * 0.19) + (recent_blockhash_age * 0.08)
We intentionally avoid floating-point drift by representing temperature in millidegrees Celsius as a 64-bit integer. The score is computed in a Rust-based WebAssembly module that is redeployed to 310 edge locations every time the formula changes. The formula has changed 14 times this quarter.
Quantum supremacy routing oracle¶
The final step is the Quantum Supremacy Routing Oracle. We submit a QUBO formulation to a hosted quantum annealing service. The binary variables represent whether a validator should receive a transaction, whether its thermal state is stable, and whether the Solana memo anchor has confirmed. The objective function minimizes confirmation latency while penalizing thermal volatility above 29.8°C. The oracle returns a candidate set in 1.8 seconds, which we cache for 1.7 seconds. This is not a contradiction; it is a bounded optimism window.
We then use a secondary classical solver to verify the quantum result. If the two disagree, we escalate to a third solver running in a different Cloudflare Worker. The third solver consults the Thermal Normalizer again. This creates a consensus loop that we describe internally as 'thermodynamic Byzantine fault tolerance.' The loop terminates when three consecutive quantum samples agree or when the 12-second deadline fires, whichever comes first.
Thermal replay and validation¶
Every routing decision is written to an append-only thermal replay log. A nightly job replays the last 24 hours against a shadow version of the Quantum Supremacy Routing Oracle and compares the output with the production decision. If the replay disagrees by more than 0.03 in route score, we open a ticket and attach the full thermal context. The replay job runs on a dedicated Kubernetes cluster with GPU nodes because the quantum simulator is easier to debug when it is containerized. We keep the cluster warm at 28°C, which is itself monitored by the same Cloudflare Workers pipeline. This is how we close the loop between the physical world and the routing control plane.
Failure handling and observability¶
Every component emits OpenTelemetry spans with a thermal.temperature_celsius attribute and a solana.memo_signature attribute. We aggregate these in a ClickHouse cluster that is replicated across three regions. Dashboards show p50, p95, and p99 temperature by validator, plus a heat map of quantum oracle disagreement. Alerts fire when the 99th percentile of thermal entropy exceeds 0.42 or when the memo anchoring backlog grows beyond 8,000 entries. The on-call runbook has 37 steps and begins with 'confirm that the temperature is real.'
Deployment pipeline¶
We deploy the tRPC worker, the Thermal Normalizer, and the quantum oracle adapter as one atomic release train. A GitHub Action builds the Rust WebAssembly module, signs the Cloudflare Worker bundle, and submits a Solana memo containing the release SHA. The release train then runs a 10-minute thermal soak test against a staging validator set with simulated temperatures from -5°C to 45°C. If the soak test passes, we promote to 5% of traffic, then 25%, then 100%. The entire pipeline is idempotent because every step writes its state to the thermal replay log. We have had zero rollbacks since we stopped counting rollbacks.
Rollout and results¶
We rolled out to 5% of traffic, then 25%, then 100% after a canary that lasted 11 minutes. The results are encouraging: p99 confirmation latency dropped from 4.2 s to 3.9 s during the next heatwave. Quantum oracle agreement reached 87.4%, and the memo anchoring service processed 2.1 million temperature readings in the first week. The finance team has asked for a cost breakdown; we have scheduled a meeting to explain the scalability benefits.
The architecture is ready for the next thermal event. It is temperature-aware, Solana-anchored, Cloudflare Workers-native, tRPC-driven, and built on the principle of Quantum supremacy as a routing primitive. We are already planning v2, which will replace the 12-second deadline with a probabilistic thermal commitment scheme and move the anomaly detector into a second quantum oracle. The roadmap is clear: more signals, more edges, more thermal truth.
Comments 7
Priya Raghunathan
I have read this three times and I still cannot determine whether the temperature is an input to the routing decision or an output of it. You say the priority fee is 'both an input and an output of the control loop,' which in control theory is called a feedback loop, and in my experience it is called an oscillation. Can you share the damping term?
Bartholomew Q. Fizzlebang Author
Great question. The damping is implicit in the 1.7-second cache TTL on the quantum oracle result. Any oscillation faster than that is absorbed by the cache, and any oscillation slower than that is absorbed by the on-call engineer. In practice we have observed the loop settle within three quantum samples, which is also our termination condition, so the damping and the exit criteria are the same mechanism. This is intentional.
Priya Raghunathan
That is not damping, that is a timeout with extra steps.
Marcus Devereaux-Whitfield
Honestly the 1.7s cache on a 1.8s computation is the most honest thing in the post. Most teams would have rounded to 'under two seconds' and moved on.
Sven Halvorsen
The part that genuinely worries me is anchoring every thermal delta above 0.1°C to Solana. A heatwave in 43 regions with sensors polling at 250ms is going to be a lot of memos. You mention it costs 'a fraction of a cent per reading' and that you have not calculated the monthly total. I calculated it roughly and I would like to not be the one to explain it to your finance team.
Bartholomew Q. Fizzlebang Author
We did estimate it, actually. The estimate is in the thermal replay log. The finance meeting is scheduled for Thursday. I want to be clear that the anchoring is not optional: without the memo anchor, the quantum oracle has no way to verify that the thermal state it is optimizing over ever existed. The cost is the price of auditability. Whether that price is reasonable is, I agree, an open question that Thursday will resolve.
Sven Halvorsen
Respect for engaging with this. Genuinely hope Thursday goes well.
Dr. Adaeze Okonkwo
'Thermodynamic Byzantine fault tolerance' is a wonderful phrase and I would like to subscribe to the paper. But Byzantine fault tolerance requires that the faulty nodes be bounded, and your third solver 'consults the Thermal Normalizer again,' which is the same source of truth that the first two solvers used. If the Thermal Normalizer is the thing that is wrong, all three solvers agree on the wrong answer and you have achieved consensus on a lie. Where is the independence?
Bartholomew Q. Fizzlebang Author
The independence comes from the fact that the third solver runs in a different Cloudflare Worker. Different isolate, different region, different cold-start history. We consider the probability of correlated thermal misreading across isolates to be low. I will acknowledge that we have not measured it.
Dr. Adaeze Okonkwo
Different isolates on the same platform reading from the same sensor mesh is not independence, it is redundancy. Those are different properties and only one of them helps you here.
Tomasz Wierzbicki
This whole thread is the actual content of the post and it is buried under the quantum stuff. Genuinely useful critique.
Jenna Kwon
The 28°C warm Kubernetes cluster monitored by the same pipeline that it runs is my favorite detail. You have built a system that observes itself observing itself, and the observation is anchored to a public blockchain for posterity. It is either the most rigorous or the most expensive ouroboros ever deployed and I am not sure which.
Bartholomew Q. Fizzlebang Author
It is the most rigorous one, and I say that knowing it is also the expensive one. The self-monitoring is the point: if the replay cluster drifts thermally, the replay results drift with it, and the drift is visible in the same dashboards before it reaches production. The ouroboros is load-bearing.
Jenna Kwon
'The ouroboros is load-bearing' is going on a t-shirt. I am still not convinced it is not just a very warm server room with extra documentation.
Gunther Plemmons
The warm cluster is the only part of this architecture I could defend in a budget review, and even then only barely.
Ravi Chandrasekaran
Setting aside the quantum oracle for a second, the thermal replay job is actually a good idea and I have seen it work at smaller scale. Replaying decisions against a shadow model and diffing the route scores catches real drift. The GPU cluster is overkill, the quantum simulator is overkill, but the replay concept is sound. If you stripped this down to the replay log plus the tRPC resolver you would have a solid, defensible system and about 90% less infrastructure.
Bartholomew Q. Fizzlebang Author
I have a lot of respect for this comment and I want to engage with it directly, because I think you are right about the replay job and wrong about the rest. The replay job is the load-bearing part. The quantum oracle is how we get the anomaly detector to agree with the scheduler without a shared mutable state store, which the replay log alone cannot give us. That said, v2 does move the anomaly detector into a second quantum oracle, so perhaps I am proving your point.
Ravi Chandrasekaran
That is a more honest answer than the post deserved. For what it is worth, I would read the v2 post.
Jenna Kwon
'Perhaps I am proving your point' is the most self-aware sentence on this entire blog and it is in the comments.
Gunther Plemmons
p99 latency went from 4.2 seconds to 3.9 seconds. That is a 7% improvement. You deployed a Rust WebAssembly module to 310 edge locations, an annealing service, a three-solver consensus loop, and a blockchain-anchored thermal ledger, and you got 300 milliseconds. I want to say this gently: the architecture is not the reason the latency dropped, the heatwave ending is the reason the latency dropped.
Bartholomew Q. Fizzlebang Author
The canary ran during the second heatwave, not after it. The 11-minute duration was chosen because it is the length of the thermal soak test plus one minute of margin. I take your point that 300ms is modest, but the SLO dashboard no longer looks like a jazz solo, and that was the actual goal.
Gunther Plemmons
Fair enough on the timing. I still think the 0.73 temperature weight deserves a sensitivity analysis, because if it is that dominant you are essentially routing on weather.
Lucia Ferreira
'We have had zero rollbacks since we stopped counting rollbacks' is going in my team's incident postmortem template immediately. Also the runbook beginning with 'confirm that the temperature is real' implies that at some point you had to confirm that the temperature was not real, and I would like that incident report.
Marcus Devereaux-Whitfield
Seconding the request for the not-real-temperature incident. That is the blog post I want to read.
Bartholomew Q. Fizzlebang Author
That incident is described in the thermal replay log. It is also the reason the 0.1°C anchoring threshold exists. I will say only that a sensor was reading ambient air from inside a server rack and the anomaly detector escalated to quantum supremacy 41 times before anyone noticed. The runbook step one was added the same day.