Sonntag, 17. Januar 2021

Debian on WD My Cloud Home single-bay (MCH) - part 2 (updated to match latest GPL sources 7.15.0)

 In part 1 Debian Buster was installed on the Western Digital My Cloud Home. Unfortunately the default Kernel lacks quite a bit of features. Compiling our own kernel is desirable. Before we getting started to compile away, some base understanding of the device's boot process is required.

Boot considerations

 When booting into Rescue mode (pressing the Reset button on Power-up) the system seems to look for the following files on an external USB stick:

  • bluecore.audio (some sort of firmware for the RTD1295)
  • sata.uImage (Kernel)
  • rescue.sata.dtb (Kernel device tree)
  • rescue.root.sata.cpio.gz_pad.img (Root-fs - padded to 4096kBytes)

The original software runs Android. That's the reason  why you'll find 24+ partitions on the harddisk:

parted -l
Model: ATA WDC WD80EFAX-68L (scsi)
Disk /dev/sataa: 8002GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start      End           Size          File system     Name         Flags
 1      34s        2047s         2014s                         FW_TABLE     msftdata
 2      2048s      67583s        65536s                        KERNEL_A     msftdata
 3      67584s     133119s       65536s                        ROOTFS_A     msftdata
 4      133120s    198655s       65536s                        ROOTFS_B     msftdata
 5      198656s    200703s       2048s                         FDT_A        msftdata
 6      200704s    202751s       2048s                         FDT_B        msftdata
 7      202752s    210943s       8192s                         AFW_A        msftdata
 8      210944s    276479s       65536s                        KERNEL_B     msftdata
 9      276480s    342015s       65536s                        ROOTFS_GOLD  msftdata
10      342016s    344063s       2048s                         FDT_GOLD     msftdata
11      344064s    352255s       8192s                         AFW_B        msftdata
12      352256s    354303s       2048s                         BOOTCODE32   msftdata
13      354304s    356351s       2048s                         BOOTCODE64   msftdata
14      356352s    358399s       2048s                         BL31         msftdata
15      358400s    360447s       2048s                         BL32         msftdata
16      360448s    425983s       65536s                        KERNEL_GOLD  msftdata
17      425984s    434175s       8192s                         AFW_GOLD     msftdata
18      434176s    499711s       65536s        fat32           CONFIG       msftdata
19      499712s    2138111s      1638400s      ext4            SYSTEM_A     msftdata
20      2138112s   3776511s      1638400s      ext4            SYSTEM_B     msftdata
21      3776512s   5414911s      1638400s      ext4            CACHE        msftdata
22      5414912s   9609215s      4194304s      ext4            DATA         msftdata
23      9609216s   13803519s     4194304s      linux-swap(v1)  SWAP         msftdata
24      13803520s  15628053163s  15614249644s  ext4            DISKVOLUME1  msftdata
If you ran the original install from the Russian site, your bootConfig on partition 18 looks something like this:
2:B:2:;

The first column is the BOOT_STATE, the second the A or B type partitions, and the last is the number of boot attempts. Further reading on this can be done in the bootloader (cmd_boot.c) source code supplied by Western Digital. Since the install leaves most partitions untouched, setting the first to 5 would perform a factory reset if you ever wanted to go back (I have actually tried that!)

For anything permanent we should therefore choose the B marked partitions, also occasionally referenced as Rescue

The first partition is somewhat special as it contains information for the bootloader from which sector to load firmware, kernel, and dtb. I believe uBoot would be able to operate on files in partitions, which would've been more hacker-friendly - in this case we have to supply a valid firmware table (map). To make matters worse there is some checksum over each referenced partition included in the map data. Hence every time a modification to a partition is made the FW_TABLE needs updating.

Besides using the obscure Russian fwtutil-1d it is probably wiser to use Silvio Gissi's fwtablectl. Here the first partition is decoded as follows:

Firmware 1: GoldKernel RO:true Compressed:false Version: 0 Size: 12727296 (12727296 with padding) Disk Offset: 184549376 (sector 360448) Load Address: 0x03000000 Checksum: 0x4b610726
Firmware 2: GoldRescueDeviceTree RO:true Compressed:false Version: 0 Size: 61199 (61440 with padding) Disk Offset: 175112192 (sector 342016) Load Address: 0x01f00000 Checksum: 0x001eb5dc
Firmware 3: GoldRescueRootfs RO:true Compressed:false Version: 0 Size: 12582912 (12582912 with padding) Disk Offset: 141557760 (sector 276480) Load Address: 0x02200000 Checksum: 0x3fff93a7
Firmware 4: GoldAudio RO:true Compressed:false Version: 0 Size: 3243056 (3243520 with padding) Disk Offset: 218103808 (sector 425984) Load Address: 0x01b00000 Checksum: 0x0dd71a22
Firmware 5: uBoot RO:true Compressed:false Version: 0 Size: 0 (0 with padding) Disk Offset: 181403648 (sector 354304) Load Address: 0x00000000 Checksum: 0x00000000
Firmware 6: Kernel RO:true Compressed:false Version: 0 Size: 12710400 (12710400 with padding) Disk Offset: 1048576 (sector 2048) Load Address: 0x03000000 Checksum: 0x4b571cf5
Firmware 7: RescueDeviceTree RO:true Compressed:false Version: 0 Size: 62778 (62976 with padding) Disk Offset: 102760448 (sector 200704) Load Address: 0x01f00000 Checksum: 0x001ff85e
Firmware 8: KernelDeviceTree RO:true Compressed:false Version: 0 Size: 62778 (62976 with padding) Disk Offset: 101711872 (sector 198656) Load Address: 0x01f00000 Checksum: 0x001ff85e
Firmware 9: RescueRootFS RO:true Compressed:false Version: 0 Size: 4194304 (4194304 with padding) Disk Offset: 68157440 (sector 133120) Load Address: 0x02200000 Checksum: 0x0df89573
Firmware 10: KernelRootFS RO:true Compressed:false Version: 0 Size: 4194304 (4194304 with padding) Disk Offset: 34603008 (sector 67584) Load Address: 0x02200000 Checksum: 0x14169514
Firmware 11: Audio RO:true Compressed:false Version: 0 Size: 3243056 (3243520 with padding) Disk Offset: 103809024 (sector 202752) Load Address: 0x01b00000 Checksum: 0x0dd71a22
Firmware 12: RescueAudio RO:true Compressed:false Version: 0 Size: 3243056 (3243520 with padding) Disk Offset: 176160768 (sector 344064) Load Address: 0x01b00000 Checksum: 0x0dd71a22
Firmware 13: RescueKernel RO:true Compressed:false Version: 0 Size: 8983912 (8984064 with padding) Disk Offset: 108003328 (sector 210944) Load Address: 0x03000000 Checksum: 0x37f0c764

Building the Kernel

In the same WD source package referenced above, a working 4.1.17 Kernel tree can be found at linux-kernel/linux-kernel/. The latest GPL sources (7.15.0) add some major improvements, including a working RTC and watchdog driver. 

I did not try to use the included compiler for cross-compilation but compiled it on the MCH instead. To get it to compile I had to install an old gcc-4.8 from Jessie

echo "deb http://archive.debian.org/debian jessie main contrib non-free" > /etc/apt/sources.list.d/jessie.list
apt-get update && apt-get install gcc-4.8
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 60 \
 --slave /usr/bin/gcc-ar gcc-ar /usr/bin/gcc-ar-4.8 \
 --slave /usr/bin/gcc-nm gcc-nm /usr/bin/gcc-nm-4.8 \
 --slave /usr/bin/gcc-ranlib gcc-ranlib /usr/bin/gcc-ranlib-4.8

Some components of the provided Kernel are very verbose by default - namely the RTC driver and the network driver. This patch comments a bunch of verbose outputs and adds the DTS changes for LED control as well as 1GB of free RAM.

(This is required assuming the GPL 7.15.0 of 2021-04-29)

cd linux-kernel/linux-kernel/
patch -p0 < kernel.patch
ln -s ../../../../../include/dt-bindings arch/arm64/boot/dts/include/dt-bindings

I also created a .config file which includes all modules to get Docker running along with NFS+CIFS filesystem modules. I did not include the rtd-1295 watchdog - see post "part 4" if you still want it.

The build process is no different than any other Kernel compile:
make menuconfig
make -j4 Image
make -j4 modules dtbs DTC_FLAGS="-p 8192"
make modules_install

Before a Kernel is made permanent in the boot FW_TABLE it is advisable to copy everything to a USB stick and see if the Kernel actually boots. (e.g. using the original install USB stick without!!! the OMV directory as we don't want to re-install)

cp arch/arm64/boot/Image /dev/sda1/sata.uImage
cp arch/arm64/boot/dts/realtek/wd-monarch-1GB.SATA.dtb /dev/sda1/rescue.sata.dtb

NOTE: in some cases I had to unplug the device for the boot to succeed - a reboot didn't get the device up! I Believe having seen some similar remarks in the community forum.

WARNING: In the comments section a user reported that with the current USB stick the disk would be wiped if the /omv directory was missing!

Installing the Kernel

To permanently install the Kernel it has to be copied to the appropriate partition along with the device tree config. The above mentioned fwtablectl will then fix the checksums in the FW_TABLE.

dd if=arch/arm64/boot/Image of=/dev/sataa8
dd if=arch/arm64/boot/dts/realtek/wd-monarch-1GB.SATA.dtb of=/dev/sataa6
fwtablectl-arm64 firmware update /dev/sataa1 RescueKernel arch/arm64/boot/Image
fwtablectl-arm64 firmware update /dev/sataa1 RescueDeviceTree arch/arm64/boot/dts/realtek/wd-monarch-1GB.SATA.dtb

Known Limitations

  • As of now I didn't get the PWM device to work that controls the LED

See my next post for how to control the LED.

49 Kommentare:

  1. Hello,

    Can we contact to talk about wd cloud? I'm trying to change kernel on wd cloud to newer one and add lvm. Need advice. Could You help me with this subject?

    Regards,

    AntwortenLöschen
    Antworten
    1. Honestly I just took the Kernel, WD provided... there have been quite some changes which are not in upstream.

      Let me know where I can try to help!

      Löschen
  2. Hi, can you please write a better tutorial for compiling the kernel (using a clean debian system) or publish your compiled kernel. I tried compiling the kernel for a few hours, but always failed.

    AntwortenLöschen
    Antworten
    1. The steps above were actually used to compile on a "clean" Debian Buster. Key is to install gcc-4.8 from Jessie. My compiled kernel is available here: https://loetzimmer.de/patches/

      Löschen
    2. Thanks, I also tried compiling on a clean debian with gcc 4.8, but it always fails with
      gcc: error: unrecognized command line option '-mlittle-endian'
      gcc: error: unrecognized command line option '-mgeneral-regs-only'

      Löschen
    3. Maybe I should've been more clear in my post, I did not cross-compile but compiled on the MCH instead.

      Löschen
  3. Great article!
    Just a novice-question: What benefits does this kernel have over the original? Are the 24+ partitions gone?
    Thanks!

    AntwortenLöschen
    Antworten
    1. Compiling your own kernel will allow you to use NFS, CIFS, Docker.

      The partition layout is more or less a requirement of the boot loader. Unless you compile and flash your own Uboot you'll have to stick to the 24 partitions.

      Löschen
    2. CAn i put your kernel (https://nerdprojekte.wordpress.com/2021/02/22/wd-my-cloud-home-to-linux-server-3-installing-a-new-kernel/) in the fox debian 9 image (https://fox-exe.ru/WDMyCloud/WDMyCloud-Home/Debian/)?

      Löschen
    3. Sure, go ahead! (It's not MY kernel at all ;) )

      Löschen
  4. Thanks for you work!
    It would be very helpful for less experienced users if your kernel would be part of the fox debian 9 image.

    AntwortenLöschen
  5. It is possible to add the virtualization so that we can use the qemu-kvm? i am able to install qemu and start windows xp but without kvm is really slow

    AntwortenLöschen
    Antworten
    1. check https://loetzimmer.de/patches/ there is an image and modules with "_virt" in the name. Try if that works for you!

      Löschen
    2. Unfortunately is not working, i put it and after the nas didn't start anymore, after fixed and installed qemu again i receive this message:

      "kvm" accelerator not found.
      No accelerator found!"

      Löschen
    3. I tried to install also aqemu because i use xfce and when i select in VM accelelrator KVM then IBM PC 64bit, i receive the error KVM KERNEL MODULE NOT LOADED

      Löschen
    4. Hello. i noticed now that in the file "MCH_modules_4.1.17 virt" there is not the directory KVM.

      Löschen
    5. Hello! I am following this subject, I downloaded the virt version too some week ago and I have this problem, I receive this error too "KVM KERNEL MODULE NOT LOADED", on the link https://downloads.wdc.com/gpl/GPL_MCH_6.6.1-123_20191114.tar.xz in the directory kernel there are those module, but it looks like are not applied on the file "MCH_modules_4.1.17_virt.tgz", i am trying to compile following the guide but I keep having errors starting already from:
      make -j4 Image
      make -j4 modules dtbs DTC_FLAGS="-p 8192"
      make modules_install
      I was able to apply a usb3 to pci-e and have other ports like wifi and vga, I installed Qemu and windows XP but unfortunately is using only one core, limited by the missing module, ok the performances are not like a normal pc, but is usefull when you can use the MCH like a substitute to the normal pc, for example now I am able to stop using the computer for converting video and audio for work using the FFMPEG on the MCH, this hardware can make things that are incredible, is slow but perfect for many other things

      Löschen
    6. Wow relly good article, compliments!!
      Please, I am becoming crazy to compile the kernel so I am able to have this module, but i receive error too, in the module archive there is not the directory virt/kvm...kvm.ko, I am not able to put this file inside the module and make it so I can have the KVM working with qemu, is there some way to make it work? I would like to use this function so I can emulate other OS and make some tests.

      Please Help!!!

      Löschen
    7. I follow your article, Good explanation!
      I was able to have a clean Debian 10 thanks to you and install also OMV, docker, etc.., I installed a docker with linux alpine, I am using much for work,(is much better now than before when it was WD software) i installed an SSD on it and now is really fast, but i would like to install windows and make the cpu "CRAZY" eheh, is there a possibility to have those KVM module on the kernel? Please!!

      Löschen
    8. I really don't have any idea why KVM modules are not built... will check again when I have some time to spare

      Löschen
    9. Omg thank you very much!!
      I also tried to find the single files kvm.ko, kvm-intel.ko and kvm-amd.ko but without success, but I find only for amd64 and I think if I find and I add I will have serious problems, maybe I lose files, so I deleted and I wait.

      On this link someone has the same problem to create modules with PI 3 https://forums.raspberrypi.com/viewtopic.php?t=232796

      First of all thanks for your help! Thanks to you, I can have docker and other new functions on it!

      Löschen
    10. Thank you Alexander for your help!
      I hope there is some possibility to enable KVM modules, in this case would be useful, to be able to use VM on MCH without limitations, so i can test different OS, I hope there is some way, on the WD source package is present. I tried to compile the module many times but i receive errors unfortunately.

      I really thank you for this biiig help!!!

      Löschen
    11. Don't shoot the messenger... ;)
      I'm afraid we'll have to live without proper KVM support. Doing some reading this would require at leat two things:
      1. a more recent Kernel - although not a guarantee that it'll work on the Realtek 1295 but 4.9.x seems to be a must
      2. some modification to the bootloader to put the CPU into nonsec mode, enabling the HYP feature

      I hate to say, but all of this is probably not worth the effort.

      Löschen
    12. Ok no problem!
      Unfortunately there are some limitation, I imagined!
      Thank you for your help!!

      Löschen
    13. OMG, no problem, I will give MCH away and I will buy a PI4, I am also tired to have those limitaion and don't have possibility to make a correct backup without problem with OMV.
      BTW: thank you Alex!

      Löschen
  6. Please, can you add the rtl8822bu wifi driver to the kernel (it is in GPL the kernel/phoenix/src/drivers/wifi)?

    AntwortenLöschen
    Antworten
    1. clarification - the driver is located here:
      \GPL_MCH_6.6.1-123_20191114\linux-kernel\phoenix\system\src\drivers\wifi\rtl8822bu\

      I redesigned everything into one partition /dev/sataa20 25GB,
      https://ds-blobs-3.cdn.devapps.ru/23944179/%C1%E5%E7%FB%EC%FF%ED%ED%FB%E9.jpg
      if you can add this driver for common devices it will be fantastic!

      Löschen
  7. Hey there!
    Is there any way to WireGuard support?
    I suppose the main thing is that there is custom kernel and no linux-headers. So, don't quite understand how to make it work.

    AntwortenLöschen
    Antworten
    1. As far as I know, Wireguard requires at least kernel 5.4.x - this is 4.1.17, long before Wireguard existed.

      Löschen
    2. Not exactly.
      Originally WireGuard was released for the Linux kernel, at least kernel 3.10 is required for installation.
      https://www.wireguard.com/build-status/

      Löschen
  8. Dieser Kommentar wurde vom Autor entfernt.

    AntwortenLöschen
  9. Reading this post I see that MCH works on RTD1295 SOC.
    I am wondering if there is a way to use hardware resources for h265 decoding.

    AntwortenLöschen
    Antworten
    1. There probably is... there is some source code of FFMPEG in the GPL download. I haven't dared to look into that - I needed a plain fast NAS, sorry ;)

      Löschen
  10. Hi Alex,
    perhaps i miss something but available memory is not 1GB after installing your kernel (from https://loetzimmer.de//patches/) :
    dd if=MCH_kernel_image_4.1.17 of=/dev/sataa8
    dd if=MCH_dtb_4.1.17 of=/dev/sataa6
    fwtablectl-arm64 firmware update /dev/sataa1 RescueKernel MCH_kernel_image_4.1.17
    fwtablectl-arm64 firmware update /dev/sataa1 RescueDeviceTree MCH_dtb_4.1.17
    then reboot
    pierre@FON31-wdnas:~$ grep MemTotal /proc/meminfo
    MemTotal: 750740 kB

    I've see that MCH_dtb_4.1.17 is 2021-07-07 and kernel build is 2021-11-15
    could this be my issue on mem avail ?
    Many thanks for your great works !

    AntwortenLöschen
    Antworten
    1. I updated the file. Thanks for noticing!

      Löschen
    2. thank you for updating, i've used your receipe to compil kernel and dtb on my side (on the mcu, like you) and installed my kernel version (same as your). i've update to debian 11 (from 10) to maximize the used space between root and /usr. Now working on uboot to gain access thru 'remote console' ont ethernet (like my seagate home with uboot) to avoid using a real serial console. will update if success. thanks Alex !

      Löschen
    3. Have a look at this for uBoot (I posted this in part 4):
      cd /
      dd if=/dev/mtd0 bs=1 count=9216 skip=98304 | tar x
      cat /tmp/factory/env.txt

      Alex

      Löschen
    4. Hi Alex, thank you i've saw it and that's exactly why i was looking near uboot :)and think about my seagate home with uboot netconsole. i've tried with virtual linux interface and wireshark but no success at this time (no packet from mcu), but i will retry this on 'real' eth, switch and so real promiscous mode. will update if found something. I also want to inspect the 'orignal android partition' to look if i can found some binaries that can interact with uboot... to have use it on my seagate home : it's clearly 'confortable' to have acces to uboot without opening the mcu (for exemple look this https://forum.doozan.com/read.php?3,14,14)

      Löschen
    5. I have used precisely that uBoot with netconsole before on other devices. I don't think WD bothered to compile that feature though.

      Löschen

  11. @Alex : please modify this :

    "Before a Kernel is made permanent in the boot FW_TABLE it is advisable to copy everything to a USB stick and see if the Kernel actually boots. (e.g. using the original install USB stick without!!! the OMV directory as we don't want to re-install)"
    With the last version of fox_exe : if you put your new kernel and dtd and suppress "omv" directory the result will be an usable system because it delete / without checking if /omv exist on the usb... (i've tested it ;) )
    Best way is to use a recovery stick and put the new kernel/dtd to check if all is ok before writing to mcu/hdd.

    AntwortenLöschen
    Antworten
    1. Ohh, that must be a different initrd then... I'll see if I post an updated initrd which I use as debugging then.

      Löschen
    2. Btw. thanks for the warning! :D

      Löschen
  12. Gibt es irgendwo ein plausibles howto für das Kompilieren vom 4.9.266 Kernel und dessen Installation auf dem My Cloud home? Bei mir läuft dieser Kernel zwar soweit ganz gut, jedoch müsste ich noch Teilen reinkompilieren. Leider finde ich nirgendwo Infos für eine generelle Installation auf der Box.

    AntwortenLöschen
    Antworten
    1. Mit den aktuellen GPL sources lässt sich in kürzester Zeit ein Kernel per Docker bauen (nicht auf der Box selbst, z.B. Docker-Desktop unter Windows):

      cd GPL_MCH_Monarch_
      cd dockerfile/
      docker build -t kdp_build_env .
      cd ../kernel
      tar xf linux-4.9.266.tgz
      # ... .config Datei kopieren oder Patches anwenden
      docker run -it --rm --privileged=true -e LOCAL_USER_ID=`id -u ${USER}` -e ARCH=arm64 -v "${PWD}:/home/user/build" kdp_build_env /bin/bash
      cd linux-4.9.266
      # von nun an kann man per "make menuconfig" den Kernel konfigurieren
      ./xbuild.sh build
      # baut den Kernel (ca. 1min auf einem Ryzen 7 3700X)
      exit
      # verlässt den Container
      cp linux-4.9.266/arch/arm64/boot/sata.uImage .
      cp linux-4.9.266/arch/arm64/boot/sata.dtb .
      cd linux-4.9.266/_install/
      tar cfz ../../sata.modules.tgz .
      cd ../..
      # kopiert alle benötigten Dateien zum anschließenden Transfer auf die Box

      Löschen
  13. Danke für die fixe Antwort!

    > cp linux-4.9.266/arch/arm64/boot/sata.uImage .
    WO ist "." - sprich, wo bin ich an dieser Stelle im Dateisystem?

    ich hab den Kernel auf der box (nativ) gemacht, dieser liegt nun in /usr/src/linux-4.9.266 ... Was bitte nun? Bei normalen Debian baue ich nun den Kernel und initrd in /boot ein, trage das danach in lilo/grub ein. Wie aber funktioniert es bei der WD-Box?

    Oben beschrieben kopiere ist uImage und dtb in das aktuelle Verzeichnis, danach werden modules ins Verzeichnis gepackt. In welches Verzeichnis oder fehlt da noch ein entscheidener Befehl? Danke.

    AntwortenLöschen
    Antworten
    1. Sorry, ich dachte, es ginge ums "bauen"... der Rest steht eigentlich schon im Beitrag unter "Installing the Kernel".

      Beim 4.9.266er Kernel liegt der DeviceTree unter arch/arm64/boot/dts/realtek/rtd129x/rtd-1295-monarch-1GB.dtb

      Das xbuild Script padded das Kernel Image noch (ist aber nach meiner Erfahrung optional):
      dd if=/dev/zero of=arch/arm64/boot/Image conv=notrunc oflag=append bs=1k count=512

      Module installierst du wie gewohnt mit "make modules_install".

      Löschen
  14. Unter dem Kernel-Install ist es für den 4.1 beschrieben, den Kernel kenne ich gar nicht. Wie ist es damit nun beim 4.9, denn ein /dev/sataaXX gibt es hier ja nicht (mehr). Hätte ich mal irgendwo ein howto finden können, so würde ich das auch selber lesen und nicht so komische Fragen stellen.....

    AntwortenLöschen
    Antworten
    1. Grundsätzlich ist die Installation für den 4.9er Kernel nicht anders als für den 4.1.17. Der Kernel und der DeviceTree müssen auf die entsprechenden Partitionen der Festplatte geschrieben werden.

      Dafür kommen Partitionen 2 und 5 (Boot A),
      8 und 6 (Boot B) oder
      16 und 10 (Boot G)
      in Frage (siehe Partitionstabelle oben).

      Wie deine der Device Name deiner Festplatte ist, weißt du wahrscheinlich selbst am besten.

      Löschen
  15. Danke sehr! Nach viel Quälerei hat es nun funktioniert, das sda21 als luks2/btrfs anzulegen. Dafür fehlte im Kernel bisher der device-mapper.

    AntwortenLöschen