ARM won the phone. It is now being sold as the inevitable future of the datacenter, the laptop and probably the toaster. The pitch is always the same: fewer instructions, lower power, better performance per watt. The fine print is always the same too: it depends on the core, the process node, the firmware, the operating system and which benchmark you chose.

There is no standard boot

An x86 server boots through a relatively boring chain of standards. An ARM board boots through whatever the vendor shipped. TF-A, U-Boot, a device tree blob, a proprietary blob in the middle that may or may not respect the memory map. There is no universal equivalent of UEFI plus SBBR that every board implements the same way. The result is that "just install Linux" means "find the vendor's fork of the vendor's fork".

The ARM boot chainreset vectorsigned blobhandoffhardware descriptionbootunexplainedROMBoot ROMTF-AU-BootDevice TreeKernelVendor BlobTrust Zone
The ARM boot chainEight stages, three vendors, one blob nobody can rebuild.

Diagram: The ARM boot chain. ROM connects to Boot ROM (reset vector). Boot ROM connects to TF-A (signed blob). TF-A connects to U-Boot (handoff). U-Boot connects to Device Tree (hardware description). Device Tree connects to Kernel (boot). Vendor Blob connects to Trust Zone and fails (unexplained).

The extension zoo

ARM is not one instruction set. It is a base plus optional extensions. NEON is optional. SVE is optional and scalable. SVE2 is a different optional. SME is optional and different again. MTE and PAC add memory tagging and pointer authentication, if the core implements them and the operating system enables them. Code that runs well on one ARM chip may simply not compile with the same flags on another.

The extension matrixoptional SIMDoptional scalableoptional againmatrixtaggingpointersnot the same codeARMv8NEONSVESVE2ARMv9SMEMTEPAC
The extension matrixEvery generation adds a box. Every box is optional.

Diagram: The extension matrix. ARMv8 connects to NEON (optional SIMD). ARMv8 connects to SVE (optional scalable). SVE connects to SVE2 (optional again). ARMv9 connects to SME (matrix). ARMv9 connects to MTE (tagging). ARMv9 connects to PAC (pointers). NEON connects to SVE and fails (not the same code).

Efficiency claims are process-node claims

A modern ARM laptop beats an older x86 laptop on performance per watt. It also uses a newer process node, a wider core and a different memory subsystem. Comparing the two and attributing the win to the instruction set is like attributing a faster car to the shape of the steering wheel. The ISA matters for decode and code density. It does not magically overcome the physics of the node you fab on.

big.LITTLE is a scheduler guess

Heterogeneous cores are a good idea. They are also a scheduling problem the operating system has to solve without knowing what your thread will do next. A thread that starts cheap and becomes expensive can find itself on the wrong core at the wrong time. The scheduler guesses. Sometimes it guesses well.

The big.LITTLE roulettewake upexpensive?cheap?throttlestoo slowtry againThreadSchedulerBig CoreLittle CoreThermal LimitLatencyRebalance
The big.LITTLE rouletteThe scheduler picks a core and hopes your workload agrees.

Diagram: The big.LITTLE roulette. Thread connects to Scheduler (wake up). Scheduler connects to Big Core (expensive?). Scheduler connects to Little Core (cheap?). Big Core connects to Thermal Limit and fails (throttles). Little Core connects to Latency and fails (too slow). Scheduler connects to Rebalance asynchronously (try again).

ARM is a good instruction set with a bad supply chain. The supply chain is the part the marketing deck leaves out.