#Content-addressed block store over NBD: why it was not done

Whether serving machine disks from a deduplicating, compressing block store through Virtualization's NBD client would save enough over templates to be worth a storage subsystem. Studied on 2026-10-07 by measuring two stopped images, probing Virtualization's NBD client with throwaway VMs and a prototype server outside the repository; no vphone machine was booted on it. Part of the disk-space study.

#Summary

Not now. Templates with new-identity clones already take most of the saving (five machines of one build: about 102 GB → 20 GB). On top of that, a block store would save roughly another 30–40%, almost all of it from two sources:

• compression (about 30%), which only a store can do for live disks;
• deduplication across builds (about 17% at 4 KiB), which matters only while several builds are kept.

That is 7–8 GB per build with one build, 25–50 GB with three builds of 5–10 machines each. The costs, several of them set by Virtualization and not by the prototype:

1. The NBD client hands the guest a write-through disk and never sends FLUSH or FUA, so a server must make every write durable before replying or accept losing acknowledged writes on a host crash.
2. The host cannot attach an NBD disk. cfw install, cfw update-environment, cfw update-kernel, the offline trim, vm rebase, snapshots, clones, export, import and the running-machine check all treat Disk.img as a file. Every offline operation would have to materialize a 20 GB raw image and write it back.
3. If the server is gone for more than about 30 s, the client gives up for good and every later guest I/O fails.
4. A garbage collector, index and map memory, log replay at start, and one more user-space hop per I/O.

#Potential, measured

Two vm clone copies of stopped machines (iPhone99,11 iOS 27.0 24A435 and an iPad on iOS 26.6.2 23G90, both booted before) were attached read-only, their System (with the orig-fs snapshot), Preboot, xART, Hardware and Update volumes mounted read-only, and every file's blocks (resource forks of decmpfs files included) mapped with F_LOG2PHYS_EXT; 343 sampled files read back identical through their physical addresses. Unmapped non-zero blocks are Data, User and container metadata. Fixed-size chunks were hashed (SHA-256, zero chunks skipped) and 1 in 32 first occurrences compressed with libcompression.

Both images together, 42 GB of non-zero data:

• Cross-build dedup is about 22% of System and 9% of Data at 4 KiB, and nearly vanishes at 16 KiB and above: identical files sit at random phases relative to the chunk grid. Aligning the grid to the container start (20,480 bytes) gave the same figures. A store would need 4 KiB chunks (a large index) or content-defined chunking.
• Identical data at the same offset between the two builds: 0.00 GB. vm rebase cannot share anything between templates of different builds; it helps only separately restored machines of one IPSW (about 8.7 GB a pair), so it does not compete with a store.
• Within one image: the iPhone's Data/User region has 1.88 GB (40% of the region) duplicating other chunks in the same region, probably copies between the Data and User volumes (unverified); System has 0.17 GB.
• Compression: System 4 KiB chunks compress to about 0.69 with LZFSE (many root file system files are already decmpfs-compressed; the dyld cache is not). Larger chunks compress slightly better (0.66 at 64 KiB) at the cost of read amplification.

#Data and User blocks

Data and User report FileVault locked on the host, and were expected to be incompressible ciphertext. They are not uniformly so. On the iPhone image about 2.0 GB of the region is compressible like plaintext (LZ4 0.55); chunks begin with SQLite format 3 (429), bplist00 (4,006), a PNG signature (874) and caff (711); 2.6 GB is high-entropy. On the iPad image, 3.5 GB compressible and 4.2 GB high-entropy; 9% of the iPad's Data chunks also occur in the iPhone's. This suggests guest file contents on virtio storage reach the disk image unencrypted while metadata and keys are protected, but it was not verified which files these are or why. See the open question.

#Projected host bytes

Template 18.6 GB never booted; 0.35 GB per clone after its first boots (3 GB for "used for a while"); 20.5 GB per separately restored machine. Store at 4 KiB with LZFSE: first template 18.6 × 0.894 (dedup within the image) × 0.69 ≈ 11.5 GB, each further build's template 18.6 × 0.81 × 0.69 ≈ 10.4 GB (likely less between adjacent builds of one device, unmeasured), each clone's divergence × 0.7 (compression only). IPSW caches not counted.

Without compression, dedup alone adds only the cross-build ~17% (about 2 GB with one build): the store's gain is mostly compression. A store with 4 KiB chunks also avoids the up to fourfold amplification of a clone diverging in 16 KiB pages (disk rebase), but a clone's first-boot divergence is only 0.35 GB.

#Virtualization's NBD client

From the SDK headers, strings of com.apple.Virtualization.VirtualMachine.xpc, an EFI probe VM with a read-only disk to watch the handshake, and a Linux guest to watch real I/O. Both probes were ad hoc signed with only the public com.apple.security.virtualization entitlement.

Guest throughput (Linux, 256 MiB disk; the NBD server kept data in the page cache, so these rows pay no durability cost and are not comparable to raw):

#Prototype and benchmarks

The prototype, in C with system libraries only and outside the repository: an NBD server (fixed newstyle; GO, INFO, EXPORT_NAME, LIST, ABORT; READ, WRITE, FLUSH, TRIM, WRITE_ZEROES, DISC, FUA honoured; one reader thread per connection, requests on a concurrent queue, FLUSH and TRIM as barriers) over either a raw file (TRIM as F_PUNCHHOLE) or a log-structured store: one append-only log of data records (SHA-256 truncated to 128 bits, LZ4 or LZFSE, stored raw unless it saves 10%) and map records (one per request, listing chunk → entry), with the hash index and map rebuilt by replaying the log. Replay stops at the first record that fails its checksum and truncates there; a map record may only reference verified data records, so recovery yields a prefix of acknowledged writes. A read-only base file passes through, so a clone is a map rather than a full import.

Import of 2 GiB sampled across an iPhone image (32 × 64 MiB): 4K + LZ4 1.55 GB at 540–800 MB/s, 4K + LZFSE 1.44 GB, 4K uncompressed 1.93 GB, 16K or 64K + LZ4 1.56 GB at 1.6–2.2 GB/s. Two writers racing to store the same new content left about 0.07% duplicate records.

Loopback, data in the page cache, 2 GiB volume:

Durable before each reply:

• The protocol and thread hops add 10–25 µs per operation. Reads from the store are as fast as raw (LZ4 decode is cheap); writes pay hashing and compression, 10–40% slower, twice that for 4K writes into 16K chunks.
• The log turns random writes into appends, so a flush takes milliseconds where a raw file takes about a second. That advantage disappears once every write must be durable, which a write-through client demands: a raw image's guest pays one F_FULLFSYNC per APFS checkpoint, a store about 1000 IOPS at QD1. Group commit (one barrier per ~1 ms batch) helps only at QD > 1.
• The log grows: about a million 4K random overwrites on a 2 GiB volume made an 8.7 GB log. Garbage collection is required, not optional.
• The first version wrote map records only on FLUSH. A guest wrote 1 MiB and synced, the server was killed and restarted, the client reconnected, and the guest read back zeros: acknowledged writes silently lost, because the client never flushes. Writing a map record before each reply fixed it. Host power loss was not tested.

A real vphone boot from NBD was not attempted: it needs a vphone-vm build with a changed attachment and therefore a new cdhash admitted by the AMFI allowlist. The Linux guest covered reads, writes, TRIM, concurrency, reconnection, the timeout and EIO on the real client.

#If it were built

• Server: not inside vphone-vm (a VM crash would take its storage with it, and one store per process cannot deduplicate across machines). One per-user helper in the bundle, started on demand, one Unix socket per machine with a path under 104 bytes. vm launch must see the socket listening first. The ~30 s client timeout bounds helper restarts, upgrades and GC pauses.
• Consistency: log-structured with replay, as the prototype; map checkpoints to bound replay; group commit; a stated contract ("after a host crash the disk returns to a consistent state a few ms old", or per-write durability at about 1/100 of the speed). APFS tolerates a consistent prefix, as it does a raw image with synchronization off.
• GC: mark from all maps and snapshot maps, copy live records into new segments, segment by segment. TRIM only clears map entries; space returns at GC, unlike a raw file where TRIM punches at once.
• Snapshots and clones: a copy of the map (125 MB dense for 64 GB at 4K, much less sparse). Attractive, but clonefile already does it for free.
• Host maintenance: the largest cost. Every offline verb becomes materialize, modify, write back, needing about 20 GB of scratch each time on hosts that are often short of space.
• Failure modes: a helper down for more than ~30 s fails every machine's disk; log corruption truncates to the last verified prefix (data goes back in time); a full host disk returns ENOSPC to the guest; one storage bug hits every machine at once.
• Upkeep: a versioned on-disk format with migrations, helper lifecycle and permissions, GC scheduling, Launchpad's size accounting, crash-injection tests, and a new minor series because Launchpad would depend on it.

#Cheaper alternatives

1. Templates, the offline trim, and deleting IPSWs (13.5 GB per build) once a template exists: all in place.
2. Compress cold templates: archive the Disk.img of a template no one clones any more (LZFSE, about 30–35% smaller by the figures above) and expand it back before the next clone. That takes the store's largest gain, compression, without touching the I/O path, host mounts or the CFW tools. Only "three or more builds all in use" misses out. Not built.
3. Cross-build dedup (~17%) needs a store or host block-range cloning, which macOS does not offer publicly. Left alone.

#When to revisit

• Disk space is still the limit with templates, trim and IPSW deletion in place, and three or more builds of ten or more machines each are kept in use at once; or
• Apple provides a host-mountable user-space block device (a DiskImages2 plug-in, or NBD the host can attach); or
• Virtualization's NBD client gains write-back with FLUSH.