One quiet evening my two-bay Netgear ReadyNAS RN102 started power cycling on its own. Boot, fan, click, boot, fan, click. I pulled the plug before the neighbours got involved and went to read up on the thing while it sulked on the shelf. The box holds the family archive, photos and documents and a decade of files, so the power cycling was not an abstract annoyance.

The reading did not improve my mood. The last OS Netgear ever shipped for this box, ReadyNAS OS 6.10.10, is a Debian 8 build on a 4.4 kernel, a base that was already elderly when it shipped. The company has since left the NAS market entirely: no updates, ReadyCLOUD and the app store both dead. My NAS is an orphan.

So, bin it. That was the sensible plan right up until it was not. The power loop turned out to be a disk and not the software at all, but I am getting ahead of myself.

The patient

Under the lid there is a Marvell Armada 370, chip 88F6710, and inside it a single Marvell PJ4B core, the house ARMv7 design. One core at roughly 1.2 GHz, VFPv3 floating point, no NEON (the kernel reads part 0x581). Beside it, 512 MB of soldered RAM, whose exact DDR part Linux keeps to itself. I did not reopen the case to squint at the chips: the disk swap that followed already gave me all the box-opening I wanted. Then 128 MB of SLC NAND and a U-Boot from 2011.

The storage is boring, which is the nicest thing you can say about storage: two 3 TB drives in mdadm RAID1, about 3 TB usable, btrfs on top. The RN104, RN202, RN204 and RN212 take the same treatment, but each one wants its own device tree blob, so I only vouch for the RN102 in front of me.

The disk that died on cue

Back to that power loop. Autopsy: one of the drives, a WD Red carrying about 65,000 pending sectors, had picked its moment to give up. Its array mate, an HGST 3 TB, held the array on its own while I went shopping.

The shopping is a hobby of its own. I keep a small automation running that watches listings, builds a price baseline from comparable ads, throws out anything whose description quietly contradicts its own title, and pings me when something honest shows up. I needed a 3 TB CMR drive, and it found a Seagate Constellation ES.3 (ST3000NM0033), new old stock on eBay, ordered and delivered without drama for under 90 EUR all in. Fixed price listing, so no sniping.

The CMR part was deliberate, because for a RAID1 rebuild under btrfs SMR’s rewrite behaviour is a trap for this workload.

The array rebuilt without any excitement at all.

A plan with an undo button

The good news was that nothing about the data layout is exotic. Any Linux can pick that array up with mdadm --assemble. Never --create, which does something that looks identical for about two seconds and then costs you a weekend. So the whole rescue rested on one rule I repeated to myself: reversible, which is not at all the same thing as safe.

The new system boots from an ordinary USB stick, with U-Boot told to prefer it and saveenv making that choice stick. NAND untouched, disks untouched. Pull the stick out and you are back to factory. The target was Debian stable (12 at the time), a bodhi rootfs (a ready-made Marvell rootfs), an LTS 6.12 kernel and the Armada 370 device tree blob.

When U-Boot decides not to talk to you, the only way in is the serial console: a 3.3 V USB-UART dongle on the four pin header hiding under the rear sticker, TX, RX and GND only, VCC left unconnected, never 5 V, 115200 8N1. The helper in the repo even catches the boot countdown for you, which matters more than it sounds. Full recipe: github.com/fabien0008/readynas-debian-migration.

Cutting the cord

When the moment came, the box switched to its Debian 12 system, still booting off the stick. Then came the part I had been putting off.

Before anything wrote to NAND I dumped every partition of it, mtd0 through mtd4, because flashing is how boxes become paperweights. A word on that, since the repo invites people to follow along: the dumps only rescue you if you can write them back, which needs a U-Boot that still boots and a serial console to talk to it, and they are only useful if they live somewhere other than the box being flashed. Mine sit on another machine.

Only then did the kernel and initrd move into NAND. The OS itself lives on its own small mirrored pair of partitions on the two disks (ext4, a separate RAID1 from the data array), and the data volume stays on the big btrfs array. Same two disks, two jobs.

It booted. The stick is retired. The box has come up on its own ever since.

The gremlins that came with Debian

Once it was actually mine to run, the box showed its personality.

The new system’s routine went like this: hope, four minutes, reboot. Hope, four minutes, reboot. I blamed the rootfs. Then the kernel. Then, briefly, myself in a more general sense.

It was the watchdog. This board has a hardware one, orion_wdt, with a 229 second timeout and no magic close, and it reboots the machine if nothing feeds it. A healthy system feeds it without ever mentioning it, which is exactly why it was the last thing I thought of. An early init script now pets /dev/watchdog every 20 seconds, well inside the 229 second budget, so a genuine hang still earns its reboot on time.

Gremlin two was the clock, which loses about 8.35 percent per day. Not drift. A commitment. adjtimex sorted it out, tick from 10000 to 10910, residual around one part per million. A small init script reapplies it at every boot, so reboots are fine; recalibrate after a kernel rebuild, since the right tick can shift with the kernel.

Gremlin three has no fix, and it is the one that still annoys me. Wake-on-LAN is off the table on this box under any OS I can build: powering it down cuts power to the network chip, and the vendor’s one watt soft-off state, the only thing that ever kept the PHY alive, is not something I can reproduce from Debian.

The disks that would not sleep

The disks then refused to spin down, and it bothered me out of all proportion. I had a twenty minute idle timer set, and a completely hands off hour still showed them awake. Time to find out what was waking them.

Suspect one: smartd, keeping its state and attribute logs on the data array itself. The writes were irregular, landing right around the twenty minute mark, so the idle countdown kept getting reset just before it could finish. Exiled the files to tmpfs.

The disks kept spinning. Suspect two: nmbd, the Samba name service, dropping its browse cache on the same array. Same exile, same shrug.

And now the trail got interesting, because with those two gone the array showed literally zero writes while still refusing to park. /proc/diskstats said nothing was happening, yet the drives stayed awake. That mismatch was the clue. Suspect three was my own metrics daemon, asking smartctl about the drives every sixty seconds. SMART reads never appear in /proc/diskstats, so the disks looked perfectly idle in every place I was looking while something I had built was resetting their sleep countdown every minute. No drive is going to find twenty minutes of peace when it gets poked once a minute. It now asks every thirty minutes, with the flag that skips drives already parked.

They now park after twenty idle minutes and stay parked, because the OS writes nothing to them while idle: logs and journald live in RAM. The box runs on SSH keys only, serves Samba and NFS with the passwords carried over through passdb.tdb, and is watched by node_exporter feeding VictoriaMetrics into Grafana. A fan daemon minds the temperature sensor, zram absorbs the memory pressure.

The checksum that was not the culprit

One thing kept nagging. Reads on my box topped out around 30 to 40 MB/s, while some community reports claimed up to 90 for the same model, on other disks and other firmware. My own measurement is my baseline, and it was too slow. A single 1.2 GHz core is not a lot of core, granted, but that gap was too big to hand wave.

I started with the most suspicious thing I could find. btrfs checksums its metadata on every read, and that metadata checksumming is mandatory: no mount option turns it off, unlike data checksums which you can disable at mount time. On this core, computing that checksum is real work. crc32c has no hardware instruction before ARMv8, and the SoC’s crypto engine does AES and SHA but never CRC32C, so the kernel grinds out a pure software CRC on every metadata node it walks. The old ReadyNAS firmware and the community kernel ship exactly the same software crc32c, which is why no kernel swap gets you out of it.

The checksum algorithm is fixed when the filesystem is created, like the node size, and there is no live conversion. My volume was born under ReadyNAS OS years ago with crc32c, the only choice a 4.4 kernel ever had. xxhash did not land in Linux until 5.5, in 2020, long after this box was orphaned. The current kernel and btrfs-progs offer xxhash, sha256 and blake2 at mkfs time, an option this box never had.

So I measured what switching would buy. I compiled a small CRC32C benchmark, table driven exactly like the kernel’s software implementation, and ran it on the NAS’s own core next to the real XXH64. CRC32C: 93.7 MB/s. XXH64: 256.2 MB/s. Same core, same compiler, 2.7 times cheaper. That looked like a gift.

Then I checked what the gift was actually worth, because switching checksums means rebuilding the filesystem, and rebuilding means about a day of copying with no redundancy on the only copy of the family archive. Too expensive to do on a hunch, but this box had a spare partition I could borrow for ten minutes: a swap mirror sitting at zero bytes used because zram does the real work. Same device, same file, same cache drops, only the checksum algorithm changed. Reads went from 50.8 to 59.3 MB/s, writes from 31.4 to 38.5. Around 17 percent on reads, 23 on writes. Real, reproducible, and nowhere near the 2.7 times the isolated benchmark promised.

The isolated benchmark measured checksumming in isolation, and checksumming is fast. Inside btrfs, it is only a fraction of what the single core does on a read: it also walks the metadata tree, moves pages through the cache, pushes data through the block layer. None of that cares which checksum you picked, so none of it speeds up. Which is the whole lesson in miniature: the isolated number promised 2.7 times, the in-place test delivered 1.17, and the difference between the two told me more about where the time actually goes than either did alone.

So I abandoned the migration. Seventeen percent is not nothing, but it was going to cost a full day of copying, a service cutover, and another full resync, all with no redundancy, all for a number that was never the binding constraint anyway. The playback stutter that started this whole investigation had already been traced elsewhere, to write flush stalls during large transfers, and fixed with a couple of vm.dirty_ratio sysctls. Streaming needs 10 to 12 MB/s; the volume delivers 28 to 40. I was optimizing a number that did not need optimizing.

One note for anyone tempted to run the same experiment: the spare swap mirror I borrowed for the A/B test is faster in absolute terms than the data volume, which is fine, since only the delta between crc32c and xxhash on the same device was the question. The percentages above are that delta, not a promise about the real array.

And you: which abandoned machine are you still keeping alive? Comments are open. Be kind, I am a hobbyist, not a support desk.