Month: August 2026

Ubuntu 24.04.4 LTS Finally!

In the past few months I have been struggling to upgrade our network attached storage (NAS) server from Ubuntu 22.04 to 22.04. I must have tried more than five times, and each time results in the machine failed to boot and I ended up having to perform a restore which I detailed in this post.

After upgrading the motherboard’s BIOS, I thought I may be in better shape and give the upgrade another shot yesterday. Unfortunately that attempt also failed in the same way.

However, I made two significant discoveries:

  1. The /boot directory contains soft links which pointed to an old kernel that no longer exists. Both files, vmlinuz and initrd.img had stale or invalid links.
  2. The second is that the machine can boot from my backup drive, which was an external SSD drive. For some reason, the upgrade process installed the boot sector on the backup drive.

Today, I tried again. This time I unplugged the backup drive during the upgrade process. I was pretty confident that this will work. BUT it failed again with the same symptoms.

I was at a lost, and decided to consult GROK (the AI). With its help, this is what I ended up doing.

I first totally deleted the boot partition and recreated the partition with the boot flag set. I used the parted command for this.

I then do the following:

# Install the efibootmgr to see boot selection
apt install efibootmgr

# Below is just a sample, actual output will be different
efibootmgr -v
BootCurrent: 0003
Timeout: 1 seconds
BootOrder: 0003,0004
Boot0003* Ubuntu        HD(1,GPT,8fc3aecc-c064-4c3c-a542-ffc866421758,0x800,0x219800)/File(\EFI\UBUNTU\SHIMX64.EFI)
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 ....
Boot0004* ubuntu        HD(1,GPT,4beb0c04-1f05-4df5-a38a-6a52caba49e2,0x800,0x219800)/File(\EFI\UBUNTU\SHIMX64.EFI)0000424f
      dp: 04 01 2a 00 01 00 00 00 00 08 00 00 00 00 00 ....
    data: 00 00 42 4f

# Mount the root fs and boot partition
mount /dev/nvme1n1p2 /mnt
mount /dev/nvme1n1p1 /mnt/boot/efi

# Mount the required system files in preparation for chroot
for i in /dev /dev/pts /proc /sys /run; do mount -B $i /mnt$i; done
mount -t efivarfs efivarfs /mnt/sys/firmware/efi/efivars
chroot /mnt

# Manually clean up the /boot directory with the bad softlinks
# Reinstall the kernel and the headers (to be doubly sure)
apt update
apt install libproc-processtable-perl
apt install linux-generic linux-headers-generic

# We then force a re-install of grub packages before installing grub
apt install --reinstall grub-efi-amd64 grub-efi-amd64-signed shim-signed
update-initramfs -u -k all
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=ubuntu --recheck
update-grub

# A quick test to see the boot entry exists and everything is in order
efibootmgr -v
exit
 
# Exit and reboot
umount -l /mnt
reboot

I was ecstatic, when the reboot actually worked!

However, there were still a few things that we had to do. The ZFS file system did not come online. I had to import it manually with:

zpool import -a
mount -a

The above resurrected the ZFS file system where ALL of my data resided. I then performed another reboot.

The last issue was that the multipathd.service did not start correctly. Apparently this was due to a library renaming change from 22.04 to 24.04. I had to do this:

sudo ln -s /usr/lib/x86_64-linux-gnu/libaio.so.1t64 \
           /usr/lib/x86_64-linux-gnu/libaio.so.1
ldd /sbin/multipathd   # libaio.so.1 should resolve
sudo systemctl daemon-reload
sudo systemctl start multipathd
sudo systemctl status multipathd

I performed another reboot and the system with all of its services are alive and well.

I then ran into a security issue. The new version 24.04 beefed up the apparmor security behaviour, which resulted in none of my local web sites working. I had to edit the /etc/apparmor.d/local/php-fpm file. The file now contains the following content:

# ZFS site tree used by Apache / PHP-FPM
/mnt/airvideo/ r,
/mnt/airvideo/** r,
/home/kang/log/** rw,

# this app writes/deletes cache files here
/mnt/airvideo/Sites/vault/prod/tmp/ rw,
/mnt/airvideo/Sites/vault/prod/tmp/** rw,
/mnt/airvideo/Sites/vault/dev/tmp/ rw,
/mnt/airvideo/Sites/vault/dev/tmp/** rw,

# ffprobe: inherit the php-fpm profile (ix) so it can read the same media tree
/bin/sh rix,
/bin/dash rix,
/usr/bin/sh rix,
/usr/bin/dash rix,
/usr/bin/zip rix,
/usr/bin/unzip rix,
/usr/bin/ffprobe rix,
/usr/bin/ffmpeg rix,

The above gave php-fpm the ability to read and write certain assets on my file system as well as have the ability to execute certain binaries. I had to activate the above rules with:

sudo apparmor_parser -r /etc/apparmor.d/php-fpm
sudo systemctl reload php8.3-fpm

After all this the server is back up and running. One final sanity test is that I wanted to ensure that this time we actually booted from the right drive.

sudo efibootmgr -v
BootCurrent: 0003
Timeout: 1 seconds
BootOrder: 0003,0004
Boot0003* Ubuntu        HD(1,GPT,8fc3aecc-c064-4c3c-a542-ffc866421758,0x800,0x219800)/File(\EFI\UBUNTU\SHIMX64.EFI)
      dp: 04 01 2a 00...
Boot0004* ubuntu        HD(1,GPT,4beb0c04-1f05-4df5-a38a-6a52caba49e2,0x800,0x219800)/File(\EFI\UBUNTU\SHIMX64.EFI)0000424f
      dp: 04 01 2a 00...
    data: 00 00 42 4f

ls -l /dev/disk/by-partuuid/8fc3aecc*
lrwxrwxrwx 1 root root 15 Aug 28 10:50 /dev/disk/by-partuuid/8fc3aecc-c064-4c3c-a542-ffc866421758 -> ../../nvme1n1p1

We will keep a close eye on it to ensure proper operation. We may not be out of the woods yet.