{"id":504,"date":"2026-09-04T13:48:47","date_gmt":"2026-09-04T12:48:47","guid":{"rendered":"https:\/\/www.fabidouille.com\/?p=504"},"modified":"2026-10-19T14:27:25","modified_gmt":"2026-10-19T13:27:25","slug":"renassance","status":"publish","type":"post","link":"https:\/\/www.fabidouille.com\/?p=504","title":{"rendered":"ReNASsance"},"content":{"rendered":"<p><em><strong>UPDATE 2026-10-19:<\/strong> Wake-on-LAN now works on this NAS, see <a href=\"https:\/\/www.fabidouille.com\/?p=592\">ReNASsance UPDATE: it wakes up<\/a>.<\/em><\/p>\n<p>Boot, fan, click. Boot, fan, click. One evening my ReadyNAS RN102 started rebooting in a loop, with all the family photos inside.<\/p>\n<p>The cause was a dead disk. But looking at the box, I found something worse: its software was dead too. So it now runs Debian 12. It also serves files twice as fast as before, and its disks finally sleep. It took a few detours.<\/p>\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/www.fabidouille.com\/wp-content\/uploads\/2026\/09\/nas-front-back.jpg\" alt=\"The ReadyNAS RN102 on its shelf: the front with power button, activity and disk LEDs and a USB port; the back with the large fan grille, Ethernet, two USB 3.0 ports and the 12 V input\"\/><figcaption>The NAS, front and back. The back is mostly fan.<\/figcaption><\/figure>\n<h2 class=\"wp-block-heading\">The hardware<\/h2>\n<ul class=\"wp-block-list\">\n<li>Marvell Armada 370: one ARMv7 core at 1.2 GHz, no NEON<\/li>\n<li>512 MB RAM, 128 MB NAND, a U-Boot from 2011<\/li>\n<li>2 x 3 TB in RAID1, btrfs on top<\/li>\n<\/ul>\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/www.fabidouille.com\/wp-content\/uploads\/2026\/09\/patient.png\" alt=\"Terminal: \/proc\/cpuinfo showing ARMv7, CPU part 0x581, no NEON; \/proc\/mdstat showing the RAID1 array [UU]\"\/><\/figure>\n<p>The last Netgear firmware is Debian 8 with a 4.4 kernel, and Netgear does not make NAS anymore. No more updates.<\/p>\n<h2 class=\"wp-block-heading\">The loop was a disk<\/h2>\n<p>One WD Red had 65,000 pending sectors. I replaced it with a new old stock Seagate Constellation ES.3, a CMR disk (SMR disks and RAID rebuilds do not go well together), for less than 90 EUR. The rebuild went fine.<\/p>\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/www.fabidouille.com\/wp-content\/uploads\/2026\/09\/wdred.png\" alt=\"smartctl on the pulled WD Red: Current_Pending_Sector 65197\"\/><\/figure>\n<h2 class=\"wp-block-heading\">Debian, without breaking anything<\/h2>\n<p>One rule: every step can be undone.<\/p>\n<p>1. Boot Debian from a USB stick. NAND and disks untouched. 2. Mount the RAID with <code>mdadm --assemble<\/code>. Never <code>--create<\/code>. 3. Copy the whole NAND (mtd0 to mtd4) to another machine. 4. Only then, put the kernel in NAND. No more USB stick.<\/p>\n<p>You need the serial console: a 3.3 V USB-UART adapter on the 4-pin header behind the sticker at the back. TX, RX, GND. Never 5 V.<\/p>\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/www.fabidouille.com\/wp-content\/uploads\/2026\/09\/serial-console.jpg\" alt=\"Left: the four pin serial header on the back of the RN102, next to the Ethernet jack. Right: a red FT232R USB-UART board with jumper wires on TX, RX and GND\"\/><figcaption>The serial header behind the sticker, and the FT232R adapter.<\/figcaption><\/figure>\n<p>Full recipe: github.com\/fabien0008\/readynas-debian-migration<\/p>\n<h2 class=\"wp-block-heading\">Small problems<\/h2>\n<ul class=\"wp-block-list\">\n<li>A reboot every 4 minutes: the hardware watchdog (<code>orion_wdt<\/code>, 229 s timeout, no \u00ab\u00a0magic close\u00a0\u00bb), and nothing feeding it. A script now writes to <code>\/dev\/watchdog<\/code> every 20 s.<\/li>\n<li>The clock loses 8.35 % per day. Too much for NTP alone. Fixed with <code>adjtimex<\/code>, tick 10000 \u2192 10910, and a script that sets it again at each boot.<\/li>\n<li>Wake-on-LAN after power-off: still not working.<\/li>\n<\/ul>\n<p><strong>UPDATE 2026-10-19:<\/strong> Wake-on-LAN works now. It was one register of the network chip, not the hardware: <a href=\"https:\/\/www.fabidouille.com\/?p=592\">ReNASsance UPDATE: it wakes up<\/a>.<\/p>\n<h2 class=\"wp-block-heading\">Disks that would not sleep<\/h2>\n<p>Three causes, each one hiding the next one:<\/p>\n<p>1. smartd writing its state on the disks \u2192 moved to RAM. 2. Samba&rsquo;s nmbd, same thing \u2192 moved to RAM. 3. My own metrics script, calling smartctl every minute. This does not show in <code>\/proc\/diskstats<\/code>! So the problem was me. Now every 30 minutes, and it skips sleeping disks.<\/p>\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/www.fabidouille.com\/wp-content\/uploads\/2026\/09\/spindown.png\" alt=\"Terminal: hdparm -C shows sdb in standby after the fixes\"\/><\/figure>\n<p>Then Grafana said: disks awake 62 % of the time. The cause was the scrub cron line:<\/p>\n<pre class=\"wp-block-code\"><code>0 4 1-7 * 0<\/code><\/pre>\n<p>\u00ab\u00a0First Sunday of the month\u00a0\u00bb? Rat\u00e9. Days 1 to 7 OR every Sunday: a 6.5 hour scrub, 11 times a month. After the fix: 62 % \u2192 27 %.<\/p>\n<p>Two more jobs were waking the disks for nothing: a weekly SMART long test (461 minutes on these drives) and <code>e2scrub_all<\/code> every day, on a box with no ext4 to scrub. Each useless wake-up costs a full 20-minute spin.<\/p>\n<p>The rest comes from the OS itself, which is on the same disks: any SSH login or cron job reads <code>\/<\/code> and wakes both disks. Once, they did not sleep for 53 hours. Next step: move the OS to its own small disk.<\/p>\n<h2 class=\"wp-block-heading\">Slow, then fast<\/h2>\n<p>Reads over Samba: 21 MB\/s, on a gigabit network.<\/p>\n<p>First, eBPF: impossible on this kernel (no kprobes, no BTF). The hardware counters are also wrong: <code>instructions<\/code> gave 2.78 million for a 100 MB <code>dd<\/code>. What works is <code>perf<\/code> with the software clock:<\/p>\n<pre class=\"wp-block-code\"><code>perf record -a -g -e cpu-clock -F 199 -- sleep 40<\/code><\/pre>\n<p>The box spends 99.5 % of its time in the kernel during a transfer. So kernel stacks are enough. First, the speed at each layer (caches dropped each time):<\/p>\n<ul class=\"wp-block-list\">\n<li>one raw disk: 92 MB\/s<\/li>\n<li>the RAID1 device: 128 MB\/s<\/li>\n<li>a file through btrfs: 49 MB\/s<\/li>\n<li>the same file through Samba: 21 to 24 MB\/s<\/li>\n<\/ul>\n<p>The RAID gives 128, btrfs keeps only 49. The profile shows why:<\/p>\n<pre class=\"wp-block-code\"><code>54.93%  [k] crc32c\n        ---btrfs_csum_update\n           btrfs_calculate_block_csum_pages\n           btrfs_data_csum_ok\n           btrfs_check_read_bio<\/code><\/pre>\n<p>btrfs checks the CRC32C of every data block it reads, in software, on one 1.2 GHz core: about 13.5 cycles per byte. This CPU has no CRC32 instruction (that came with ARMv8) and no NEON. And the kernel&rsquo;s <code>crc32c<\/code> function is only 60 bytes of code: the simple loop that reads one byte at a time from a 256-entry table.<\/p>\n<h3 class=\"wp-block-heading\">Samba: one line<\/h3>\n<p>In the same profile, 7 % of the CPU was <code>arm_copy_from_user<\/code> under <code>tcp_sendmsg<\/code>: smbd copying file data into the socket. <code>use sendfile = yes<\/code> is supposed to remove this copy, and it was already in my smb.conf. But since Samba 4.6, <code>aio read size<\/code> is 1 by default, and Samba never uses sendfile when AIO is on:<\/p>\n<pre class=\"wp-block-code\"><code>aio read size = 0<\/code><\/pre>\n<p>Samba: 21-24 \u2192 31-34 MB\/s. Two measures each side, because one measure proves nothing.<\/p>\n<h3 class=\"wp-block-heading\">A faster CRC, same checksums<\/h3>\n<p>I measured the options on the NAS itself, 120 MB from the page cache:<\/p>\n<ul class=\"wp-block-list\">\n<li>CRC32C slice-by-8: 144.7 MB\/s<\/li>\n<li>CRC32C slice-by-4: 133.4 MB\/s<\/li>\n<li>CRC32C one byte at a time (the kernel): 93.5 MB\/s<\/li>\n<li>xxhash64: 69.6 MB\/s<\/li>\n<li>sha256: 30.5 MB\/s<\/li>\n<li>blake2b: 23.2 MB\/s<\/li>\n<\/ul>\n<p>Slice-by-8 reads 8 bytes per loop with 8 tables. Same polynomial, same result, so nothing to reformat. I hoped for 3 to 4 times faster. Measured: 1.55. (Slice-by-16 is 3 % slower: too many tables for the 32 KB L1 cache.)<\/p>\n<p>xxhash, my first idea, is slower than the simplest CRC on a 32-bit CPU. Lucky that I did not reformat 2 TB for it.<\/p>\n<p>The SoC has a crypto engine (CESA), but it cannot do CRC, and its SHA-256 is 2.8 times slower than the software CRC on 4 KB blocks.<\/p>\n<p>A wrong CRC makes the whole volume look corrupted, so I tested before building: the standard check value <code>CRC32C(\"123456789\") = 0xE3069283<\/code>, then 32,789 comparisons with the reference code, and a self-test that refuses to load if one result differs.<\/p>\n<p>Loading it as a module changed nothing: in this kernel, btrfs calls <code>crc32c()<\/code> directly. So: a new kernel, in NAND. In the same build, <code>init_on_alloc<\/code> off: 12 % of the CPU was used to fill new pages with zeros, pages that the disk overwrites just after.<\/p>\n<h3 class=\"wp-block-heading\">The kernel that never had modules<\/h3>\n<p>Building modules, I found that no module could load on this box since the migration:<\/p>\n<pre class=\"wp-block-code\"><code>loop: disagrees about version of symbol path_put<\/code><\/pre>\n<p>The kernel in NAND and <code>\/lib\/modules<\/code> came from two different builds: my first <code>make modules<\/code> had been stopped. Everything important was built into the kernel, so nobody noticed. 2,400 modules later, <code>modprobe<\/code> works.<\/p>\n<h3 class=\"wp-block-heading\">The boot loop that was not<\/h3>\n<p>After the flash: ports closed, and the IP address changing every 15 to 25 seconds. I was sure it was a boot loop. With the serial cable, the box was simply booting: slowly (first boot with modules), and a script restarts the network during boot. Closed ports do not tell you if a box is dead or still booting. Only the serial console does.<\/p>\n<h3 class=\"wp-block-heading\">NFS<\/h3>\n<p>The NFS server <code>max_block_size<\/code> was 64 KB. Raised to 1 MB. (It refuses the change while nfsd threads exist: set them to 0, write the value, restart.)<\/p>\n<pre class=\"wp-block-code\"><code>echo 0 &gt; \/proc\/fs\/nfsd\/threads\necho 1048576 &gt; \/proc\/fs\/nfsd\/max_block_size<\/code><\/pre>\n<p>Result:<\/p>\n<ul class=\"wp-block-list\">\n<li>Samba: 21 \u2192 41 MB\/s<\/li>\n<li>NFS: 43 \u2192 65 MB\/s<\/li>\n<li>btrfs read on the box: 49 \u2192 60-66 MB\/s<\/li>\n<\/ul>\n<p>The fastest kernel did 71.8 MB\/s, but it was 6,473,835 bytes and the NAND partition holds 6,291,456. The one I flashed has 32,813 bytes free.<\/p>\n<p>90 MB\/s is possible: a small btrfs patch lets NFS read directly from disk, without the server cache (81.3 MB\/s on 4 GiB). But then the disks wake up more often. I chose sleep. Grasse matin\u00e9e for everyone.<\/p>\n<h2 class=\"wp-block-heading\">How AI was used<\/h2>\n<p>Most of the hands-on work was done with AI coding agents (Claude Code) on my home server, with SSH to the NAS: perf profiling, the slice-by-8 patch and its tests, kernel builds, the NFS tests, the Grafana history. They are fast and tireless, but not always right: the \u00ab\u00a02.7 times faster xxhash\u00a0\u00bb came from an agent comparing two different programs, and the \u00ab\u00a0boot loop\u00a0\u00bb was an agent reading too much into closed ports. My part: ask the questions (in French: \u00ab\u00a0why was it faster with the original firmware?\u00a0\u00bb), refuse answers that did not convince me, and choose: disk sleep over 90 MB\/s.<\/p>\n<p>Note: I don&rsquo;t have much time for the blog these days, so this post was written with the help of an AI, from my own notes, then read and corrected by me. Better to share the content this way than not at all.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>An old Netgear NAS with no more updates, a dead disk, and Debian: now twice as fast, and the disks finally sleep.<\/p>\n","protected":false},"author":1,"featured_media":526,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-504","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-non-classe"],"_links":{"self":[{"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/posts\/504","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=504"}],"version-history":[{"count":14,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/posts\/504\/revisions"}],"predecessor-version":[{"id":593,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/posts\/504\/revisions\/593"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=\/wp\/v2\/media\/526"}],"wp:attachment":[{"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=504"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=504"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fabidouille.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=504"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}