Experimental Machine for AI

The advent of the Large Language Model (LLM) is in full swing within the tech community since the debut of ChatGPT by openAI. Platforms such as Google Colab, and similar variants from Amazon and Facebook allows software developer to experiment with LLM’s. The hosted model of the data center based GPU’s makes training and refinement of LLM’s tolerable.

What about using LLM on a local computer away from the cloud?

Projects such as llama.cpp by Georgi Gerganov makes it possible to run the Facebook open sourced Llama 2 model on a single MacBook. The existence of llama.cpp gives hope on creating a desktop that is powerful enough to some local development with LLM’s away from the cloud. This post documents an experimental procedure in building a desktop machine using parts readily available from the Internet to see if we can do some AI development with LLM’s.

Below is a list of sourced parts from EBay, Amazon and CanadaComputers, a local computer store. All prices are in Canadian dollars and includes relevant taxes.

NVIDIA Tesla P40 24GB GDDR5 Graphics Card (sourced from EBay)$275.70
Lian-Li Case O11D Mini -X Mid Tower Black (sourced from Amazon)$168.49
GDSTIME 7530 75mm x 30mm 7cm 3in 12V DC Brushless Small Mini Blower Cooling Fan for Projector, Sleeve Bearing 2PIN (sourced from Amazon)$16.94
CORSAIR Vengeance LPX 64GB (4 x 32GB) DDR4 3200 (PC4-25600) C16 1.35V Desktop Memory – Black (sourced from Amazon)$350.28
AMD Ryzen 7 5700G 8-Core, 16-Thread Unlocked Desktop Processor with Radeon Graphics (sourced from Amazon)$281.35
Noctua NH-D15 chromax.Black, Dual-Tower CPU Cooler (140mm, Black) (sourced from Amazon)$158.14
Asus AM4 TUF Gaming X570-Plus (Wi-Fi) ATX motherboard with PCIe 4.0, dual M.2, 12+2 with Dr. MOS power stage, HDMI, DP, SATA 6Gb/s, USB 3.2 Gen 2 and Aura Sync RGB lighting (sourced from Amazon)$305.09
Samsung 970 EVO Plus 2TB NVMe M.2 Internal SSD (MZ-V7S2T0B/AM) (sourced from Amazon)$217.72
Lian Li PS SP850 850W APFC 80+ GOLD Full modular SFX Power Supply, Black (sourced from CanadaComputers)$225.99
Miscellaneous 120mm case fans and cables purchased from CanadaComputers$63.17

The total cost of the above materials is $2,062.87 CAD.

The Nvidia Tesla P40 (Pascal Architecture) specializes for Inferencing limited to INT8 based operations and does not support any FP related operations, so it may not be optimal for machine learning. However recent claims have been made that INT8 / Q8_0 quantization can yield some promising results. Let us see what our experimentation will yield once the machine is built.

A custom design 3D fan shroud has to be designed and 3D printed because the P40 does not natively come with active cooling. The P40 is originally designed to operate in a data center so cooling is provided by the server chassis. The custom shroud design is posted on Thingiverse and some photos of the finished shroud is shown below.

Note that M3 screws were used to secure the shroud to the P40 GPU card. The GDSTIME fan came with the screws.

I also made a mistake by initially getting a 1000W ATX power supply that ended not fitting the case, because the case is built for SFX and SFX-L power supplies. Lesson learned!

Once the machine is built I performed a 12 hours MemTest86+. It turned out that running the memory at the XMP profile was a bit unstable. I had to clock the memory back from its 3200MHz rating to 3000MHz.

After more than 12 hours with 3 passes.

The BIOS settings had to be configured so that Resize BAR is ON. This is required for the P40 to function properly.

Turn on Resize BAR

The next step is to install Ubuntu 22.04.3 LTS with Nvidia GPU and CUDA drivers. The latter was quite challenging. The traditional way of installing using the package manager did not work. The best way is to goto this site, and pick the run file like below:

Beside to use the runfile

The run file had to be run in recovery mode using the console because the installation will fail if an X11 window manager is running. Also all previous Nvidia drivers had to be removed and purged. The Ubuntu default installation process may have installed them.

A detail that was left out of the instructions is to set the appropriate shell paths once the installation is completed. The following changes were made with /etc/profile.d so that all users can benefit. If the login shell is using zsh, then /etc/zsh/zshenv has to be changed. Without this change, commands such as nvcc and other CUDA toolkit commands will not be found. The same is true for CUDA related share libraries.

$cat /etc/profile.d/cuda-path.sh

export CUDA_HOME="/usr/local/cuda"

if [[ ! ${PATH} =~ .*cuda/bin.* ]]
    export PATH="${PATH}:/usr/local/cuda/bin"

if [[ ! ${LD_LIBRARY_PATH} =~ .*cuda/lib64.* ]]
    export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/cuda/lib64"

if [[ ! ${LD_LIBRARY_PATH} =~ .*/usr/local/lib.* ]]
    export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib"

In this hardware configuration the AMD CPU has integrated graphics, and the P40 does not have any HDMI or DisplayPort connections. We need to change the X11 configuration so that it will only use the AMD CPU while dedicating the P40 GPU for CUDA based computation. The following configurations have to be made in /etc/X11/xorg.conf:

$cat /etc/X11/xorg.conf

Section "Device"
    Identifier      "AMD"
    Driver          "amdgpu"
    BusId           "PCI:10:0:0"

Section "Screen"
    Identifier      "AMD"
    Device          "AMD"

The BusId can be obtained using the lspci command and be sure to change any hexadecimal notations to decimal in the configuration file. Without this xorg.conf configuration, the Ubuntu desktop will not start properly.

When everything is done properly, the command nvidia-smi should show the following:

Fri Aug 25 17:33:31 2023
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|   0  Tesla P40                      Off | 00000000:01:00.0 Off |                  Off |
| N/A   22C    P8               9W / 250W |      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |

| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|  No running processes found                                                           |

The machine is now ready for user account configurations.

A quick video encoding using ffmpeg with hardware acceleration and CUDA was performed to test the GPU usage. It was a bit of a challenge to compile ffmpeg with CUDA support. This is when I found out that I was missing the PATH configurations made above.

For good measure, gpu-burn was run for an hour to ensure that the GPU is functioning correctly.

Next step is to download and setup the tool chain for LLM development. We will save that for another posting.

Short Road Trip to University of Waterloo

This morning we wanted to show our son what the University of Waterloo campus looked like. This is also an excellent opportunity to get use to our new Tesla Model Y along with its Autopilot functionality.

As the above pictures show, the campus is a ghost town, but it was nice that most of the buildings were still opened and we took the opportunity to walk around.

Aside from the typical Toronto traffic, the drive itself was pretty uneventful. While on Highway 401, I had the car on both Auto Steer and Traffic Aware Cruise Control (TACC) for more than 80% of the trip. For the most part it worked well, but I did notice a few points:

  • Passing trucks seem to be a lot closer than I like when I manually pass a semi on the highway;
  • When in Auto Steer, the car seem to initiate the turn a little later than what I would normally do; and
  • Finally, one can get really use to TACC and forget to press the brake when TACC is off;
  • I love TACC when in traffic – it is a godsend!

We also took the opportunity to do a quick test supercharge in Cambridge, Ontario at 22 Pinebush Rd, Cambridge, ON N1R 8K5.

We really did not have to charge, since our Long Range model had enough juice to go there and back. We just wanted to experience what supercharging is like. It was super simple! We added 10kWh for $3.93, and that took about 6 minutes. During this time, I showed my wife how to do it.

Another experiment that we did was I ran Waze on my iPhone in concurrent with Tesla’s navigation routing and mapping. Albeit I currently have a sample size of one, but my feeling is that Waze is still better at this point in terms of traffic awareness and routing abilities. Also Tesla maps do not show obstructions, speed traps, and other goodness of Waze. I will be doing more of this comparison on our trip to Montreal next Friday.

Tesla Delivery Day

Today is the day! We departed our house at 10:45 am and arrived at 2 Chrislea Road, Vaughan, Ontario at 11:20am to pick up our new Model Y 2023. I drove into the service centre that was literally filled with Tesla vehicles to the point where we could not park at the service centre. We had to park at the next lot and walked over.

Location of the Tesla Vaughan Service Centre

It was not too bad, as if “someone up there” has arranged to put the rain on pause, while we walked into South entrance.

The first greeting counter

At the greeting counter, I was handed some paper work and told to sign at the relevant locations, which I did. The reception was a well oiled process. After the paperwork was handed over along with my bank draft at the second counter, I was handed the “card keys”, ownership paper, and my chaperone took me to the car. They didn’t even check if I had proper insurance!

Chaperoned to my new Model Y

He helped me to pair my iPhone 14 Pro to the car. Gave me a few tips on operating the wipers, and asked me if I had any questions, which I said “no.” I got into the car and adjusted my seats and mirrors, and we were on our way! I am certain, if I had any questions, he would have been of great help. However, I followed Tesla’s advice and watched all the delivery day videos, so I knew what I had to do. He did tell me that today at this centre, they are going to move over a hundred vehicles. This is more vehicles than any other dealerships in a single month. GM better up their game plan!

I dropped my wife off to her Toyota Prius Prime, and then routed the car to drop my neighbour off to pick up his 10 years old Tesla Model S that was being worked on. My neighbour being a Tesla champion and expert came along with us and was of great guidance to us through the delivery process. Thank you Johnny!

My beautiful wife in front of the Service Centre

On the way home, I established the data link (Premium Connectivity) and turned on Auto Steer. I had to drive a little ways for the car to calibrate itself before I can turn on the cruise control and auto steer. This feature was super handy when I got stuck at a traffic jam on the way home. My new Tesla was able to follow the car in front of me while I was bumper to bumper in the jam.

Once I got home, I tested the Mobile Charger to make sure that it was working and stuck the charger in the trunk, which is where it will live and use during road trips. I added my wife as a second driver, and we added profiles for myself and my wife, and went out for a late lunch. It was very weird to go out without any keys. I can now use my iPhone for both car and home locks.

Now we have two green vehicles in our garage.

Tesla Model Y 2023 (left), and Toyota Prius Prime 2020 (right)

Sorry Subaru Impreza 2013, you will now have to park in the drive way for now.

I have scheduled the charger to kick in at 5am tomorrow so that we can take a small road trip to the University of Waterloo tomorrow. We wanted to show my second son the campus as he considers his options for September.

Tesla charging through the Wall Connector at 40A (~9kW)

One last thing, I also configured the car with my home’s WiFi, and thought it was pretty cute to have the car showing my solar power generation status using my custom power dashboard.

Tesla displaying my solar power generation stats using the in car’s browser

Tomorrow to Waterloo, Ontario, and next week we’ll be going to Montreal! Going to look forward to our Supercharging experience.

New 1.5Gbps Internet Service

On April, 4th, I received a promotional offer from Rogers offering Ignite Internet service at 1.5Gbps plus Streaming for $114.99 per month.

I procrastinated a bit because I wanted to make sure that I can actually make use of this service. However, when I checked my bill for April, I noticed that my total monthly charges is at $102.99.

Note the above price prior to discount is at $117.99. I was curious to see if Rogers can get me a good deal without the Streaming service. I called Rogers support line and received a person who was not very helpful and simply just quoted conditions and deals to me. AI will do a number of these types of people soon.

I decided to try an alternative route by using Twitter (@RogersHelp). I direct messaged Rogers on Twitter and received wonderful help. They offered me the 1.5Gbps service at only $104.99 (with a 24 months commitment). This will be somewhat on par with my current payment and I will get 50% more throughput.

There is another question. Will my networking equipment make use of the 1.5Gbps? My networking setup has the Rogers Ignite WiFi Gateway (ARRIS Group, XB7 Modem) and is connected with a Cat5e cable to my Unifi Dream Machine Pro, using one of its 1Gbps RJ45 port.

Rogers XB7 Modem Unifi Dream Machine (UDM) Pro(Firewall / Router)
Home Network
Home Network
Text is not SVG – cannot display

How can we overcome the 1Gbps limit on the UDM Pro’s RJ45 port? Luckily the UDM Pro has an 10G SFP+ port as well. I went to Amazon and purchased a 10G SFP+ RJ45 Copper Module Transceiver.

The above will auto negotiate a 2.5Gbps to 10Gbps connection from the XB7 to my UDM Pro. Of course I will not be getting 2.5G or 10G speeds. These are just the physical maximum per the respective devices. Rogers will throttle my inbound and outbound traffic to 1.5Gbps and 50Mbps respectively.

After installing the SFP+ module and rewire the existing Cat5e cable, I had to reboot the both the XB7 modem and the UDM Pro. Once everything came back up, I had another problem. How do I test that I actually get 1.5Gbps? I cannot do it from any WiFi devices or any wired devices in my house because they are all limited to the 1Gbps port speed from my networking switches. Once again, Unifi had thought of this already and provide a speed test functionality on its management dashboard.

The tested speed seems to be better than expected.

So you can see from the above screen shot, that we are now getting what we are paying for. I also performed a double test from two different machines that are routed to a switch that has a 10Gbps connection to my UDM Pro, and each machine received a 700Mbps to 800Mbps download speed, which is around 1.5Gbps in aggregate. Mission accomplished.

Unifi just came out with a new firmware update that enable the UDM Pro to perform load balancing of more than one WAN connection. When the SkyLink service becomes more economically feasible, we can attach a satellite based internet service as a compliment to the existing Rogers service. This way during a power outage, we can continue to get Internet.

Tesla Order Update

It looks like Transport Canada has updated their eligible vehicles for their Incentives for Zero-Emission Vehicles Program.

From Transport Canada

From above, it is clear that the Model Y Long Range AWD is now eligible! This is great news. I will be able to save $5000 from the purchase price.

When I checked my Tesla account, I can see that my order has been updated.

That is of course excellent news. However there is a downside. The delivery date is now pushed out to be between late July and early September, so fingers crossed that we’ll be getting it sooner rather than later.

Above estimate as of April 25th, 2023

Update May 1, 2023:

Above estimate as of May 1st, 2023

Update May 6, 2023:

Received an email update indicating “Final Payment is now ready”. Went to the Tesla site and into my account and received the bank wire info. We will call the bank on Monday to arrange the transfer.

Above is displayed as of May 6th, 2023

Update May 14, 2023:

I received a text message this morning:

I chose the 20th, and we are temporarily booked for 11:30am for delivery. I have not received a VIN number as of yet. Hope to get this soon, as I will need it for augmenting our auto insurance.

The Pursuit of a BEV

This is our journey to buy a full Battery Electric Vehicle (BEV).

When Tesla announced their Model 3 back in 2016, I was one of the first to place a $1,000 reservation for one. The promise was for an electric vehicle costing $35,000 USD. In the spring of 2018 when the first delivery to Canada happened, the price was $64,100 CAD for the Long Range Model 3. This is not the AWD version, and with the taxes it will exceed $70K. The on the road – all in – price from one of our neighbours exceeded $80K when his vehicle got delivered. Okay, at the time there was a $15K incentive from the government.

The final sticker shock was a bit of a surprise, and we were still probably not ready for a full electric vehicle at the time. We checked out the Nissan Leaf, the Hyundai Kona, and the Kia Niro. These were all in high demand with waiting lists exceeding a year. We waited for the Subaru Impreza Hybrid, which never made it to Ontario, and the availability in Quebec was spotty at best.

In a moment of pure coincidence, we got hold of a Toyota Prius Prime on October, 2019. This is our first Plugin Hybrid Electric Vehicle (PHEV). We decided on a PHEV to get our feet wet with EV tech but also mitigated our fear of range anxiety. We felt less stress with a hybrid, and the small plugin battery which offers a limited range of the 35 to 45km. This short range will make do for 90% of our trips, which are mainly grocery runs, errands, and trips to local restaurants. We thought this PHEV will be perfect for us.

The experience driving in EV mode with the Toyota has made gas station visits an extreme rarity. We are talking like two partial fill ups during 2020, and probably less than 10 fill ups to date (all partials), and we are on its fourth year driving the Toyota.

We love the PHEV experience so much that on July 27, 2021, we placed a reservation on a RAV4 Prime at Richmond Hill Toyota and were warned that the wait will be very long.

Hedging our bets, we later placed another reservation on a KIA EV6 at KIA Stouffville on September 2, 2022. We thought the EV6 would be in a similar price range of the RAV4 Prime. You can read more about our reservation experience here.

On March 7, 2023, I received a call from my contact at Richmond Hill Toyota and he told me he had a 2023 RAV4 Prime XSE available. The price was $66,073 CAD all in. In January, Tesla dropped their price on the Model Y Long Range AWD to $69,900 CAD (from $85,000). Since we already have a fossil car, a 2013 Subaru Impreza; and a PHEV, the 2020 Toyota Prius Prime, we decided to pull the trigger and reserved the Tesla Model Y on the same day. Below is the configuration and the price breakdown.

Basic configuration (click to enlarge)
Price Details (click to enlarge)

The $81K price tag is higher than the $66K of the RAV4, but I figured that I will probably not buy another car for a very long time. At my age, I minus will cease the waiting and enjoy what life remains. Assuming Tesla meet its commitment on delivery timeframes, we should be getting the car before July, fingers crossed!

I also pulled the trigger in getting the mobile charger, wall connector, and the All-Weather Interior Liners. As a matter of fact, the interior liners already arrived and the charge accessories have already been shipped. We will hit the ground running (or driving). In the meantime, we are back in the waiting game again. May the EV gods be kind to us.

I am already super addicted to YouTube channels that are sharing other people’s Tesla experiences. My wife and I are already quite excited and may even partake on a few road trips with the new purchase. Perhaps testing out the Supercharging network to Montreal, and then an across Canada trip from Toronto to Calgary and perhaps even to Vancouver. We will see.

I will update with another post when the car arrives!

Playing with Proxmox

Prior to the holidays in 2022, I upgraded my media NAS server as detailed here. After this upgrade, I repurposed the old server’s components and built another PC.

Originally I was going to use this extra PC as a simple online media encoder, since encoding videos in the HEVC codec takes a lot of CPU power. I did this for about a month. My son, Kalen had an old GTX1060 6GB graphics card that he was going to place on Kijiji for resale. I offered to purchase this graphics card off of him so that that I can pair it up with this repurposed PC. The new idea was to turn this PC into my gaming PC. I don’t do many 3D intensive gaming, so an older GPU is certainly good enough for me.

Off I went installing Windows 10 Pro on the PC. I also discovered at this time the Windows Subsystem for Linux (WSL). I thought it would be a wonderful idea for me to have the gaming PC and not lose the ability for the PC to double as a media encoder through the use of a Linux distribution using WSL. My hope is that Linux with WSL will yield near metal based performance. Long story short, the performance of ffmpeg, the tool that I use for video encoding, was disappointing. Apparently there is a bug in WSL v2 that forced ffmpeg to only use 50% of its CPU power. There was nothing wrong with the concept of having a dual purpose PC for gaming and a handy Linux distribution for other endeavours.

The problem is with the Windows hosted Hypervisor, a software layer that usually runs between the hardware and the operating. I know of another hypervisor called Proxmox. This is a perfect opportunity for me to try Proxmox out. Before I installed Proxmox, I maxed out the memory of this repurposed PC to 64GB. It only had 16GB before and I thought this would not be enough.

One of the worries I had was how to get the raw GPU performance from Proxmox? Apparently there is a GPU passthrough option. Before installing Proxmox, I had to make some BIOS adjustments on the PC.

  • Enable IOMMU
  • Enable SVM Mode (same as Intel VT-x)
  • Enable AMD ACS

Only the SVM Mode is required for Proxmox, the other two are required for GPU Passthrough. After I installed the Proxmox server, I followed the instructions outlined in the following sites:

  1. From 3os.org: GPU Passthrough to VM;
  2. From pve.proxmox.com;
  3. And from reddit.

The first site was more clear and was the most helpful. I used the second and third sites as an alternate source and backup reference. Thanks to the above sites, I was able to get Proxmox running and created two virtual machines (VM’s). The first is an Ubuntu distribution called workervm and the second is a Windows 10 Pro instance with a GPU passthrough, called win10. Below is a screenshot of the Proxmox control administration site.

Proxmox control panel (click to enlarge)

Below is the workervm (Linux VM) configuration:

workervm configuration for Ubuntu instance

I had to make sure the processor type is set to [host] to get the most performance out of the virtual CPU’s. The Windows VM configuration uses a different BIOS, specifically a UEFI BIOS. We also have to ensure that the Machine type is set to q35. The Windows VM also has the EFI Disk and TPM State configured, and of course the extra PCI Device to represent our GPU passthrough card. Check out the full configuration for the Windows 10 VM below:

win10 configuration for Windows 10 Pro instance

After installing Windows 10 Pro, the network interface is not recognized. To remedy this situation I had to install virtio-win as described by this site here. After the installation of virtio-win, and a reboot. I had networking, connectivity to the Internet, and the Device Manager output from the Windows 10 Pro instance as shown below. Notice that Windows recognized the native NVIDIA GeForce GTX 1060 6GB card.

Windows 10 Pro VM instance Device Manager

I tried to test out the GPU throughput with some 3D rendering demos and tested a couple of games from Steam using Remote Desktop. The performance was okay, but not stellar, and could have been better. I did some more research, and apparently Parsec, a virtual desktop sharing tool, is probably better for remote gaming.

I went ahead and installed Parsec on both the Windows 10 Pro VM, and on my Mac mini, which I used to remotely play games on the previous VM. This worked out to be quite well.

Now the repurpose PC is a Proxmox server hosting as many VM’s that the hardware can bear. The workervm instance can be used for video encoding and other generic Linux oriented work or trials. The win10 instance will be used for gaming and hosting our tax filing software, called TurboTax, which only runs on Windows.

In the near future, I will also be testing out Proxmox with virtual containers instead of machines. The containers are more light weight and less resource intensive. It will be another new adventure here.

Panel Snow Coverage

Today is January 13, 2023. We had an icy snow storm last night that lasted until this morning, and I was curious what the roof condition was like. Just how much of the panels were covered in snow?

Solar energy for today

Our peak energy production was at around 11am when we generated a little over 800Wh, which is inlined with what we kind of get on a cloudy, misty, winter day. In contrast, the best we got so far was on January 7th at 1pm. We generated 5,494Wh. That was a sunny day with no snow coverage on the panels.

A quick drone survey of our roof this afternoon at around 3pm.

I was kind of impressed that we got that much with so much of the panels covered. Watch the above video to see just how much of the panels are covered today. Our total production for today is only about 3,400Wh.

Below are the stats per panel.

Per panel generation statistics for today.

As you can see above, every panel contributed even the covered ones! There will be two sunny days over the weekend, so we will see!

Update: 2023-01-14

I did another roof survey with my drone, seeing that today it was a sunny sky day.

Roof survey on Jan. 14 (day after storm)
Solar energy production on Jan. 14

We have generated over 10,000 Wh of energy today about 3 times more than yesterday. The survey was conducted when it was still -6 ºC outside, so way below freezing.

Media Server Upgrade 2022 (Part 2)

Part 1

In the first part of this post, I talked about making sure all the new hardware that I recently purchased works. Yesterday, upgrading from Ubuntu 20.04 LTS to 22.04 LTS was super simple. Unfortunately, that was the end of the easy part.

I thought I could just image by old boot drive and make a carbon copy of it on my new boot drive. My old boot drive is a simple SATA 512GB SSD, and my new boot drive is an NVMe M.2 1TB SSD plugged directly to the motherboard. The copying was pretty simple, but because the drives differ in size, I had to relayout the partition table with the new drive once the copy is completed. I did this with the parted command.

Unfortunately the new boot drive did not want to boot. At this point I had to do some research. The most helpful articles were:

Both of the above articles were an excellent refresher on how GRUB works. I have used GRUB since the beginning, but one gets super rusty when these types of tasks are only performed once every three or six years!

Instead of detailing what went wrong, I will just explain what I should have done. This way if I need it again in the future, it is here for my reference.

Step 1: Perform a backup of the old boot drive from a Live USB in shell mode. This is done on my server on a nightly basis. This method is clearly described on the Ubuntu Community Help Wiki.

Following this method I will end up with a compressed tar archive for my entire root directory, skipping some runtime and other unwanted directories.

Step 2: After installing a fresh install of the new Ubuntu LTS Server operating system on the new server and boot drive, I proceeded to backup the new boot with the same technique used in Step 1. I stored the backup of the new install on another external SSD drive that I have lying around. Also it is important that new boot drive partition layout of the new install contains a swap partition.

Step 3: I then restore the most recent backup (done in Step 1) of the old boot drive to the new boot drive. I then replaced the /boot/grub directory with the new contents from the new install which was backed up in Step 2. The new GRUB is already installed when we performed a brand new installation on the drive. We just want to make sure the boot partition matches the /boot/grub contents.

Step 4: We also need to fix up the /etc/fstab file because it contains references to drive devices from the old hardware. Paid special attention the main data partition and the swap partition. It should look something like this:

# /etc/fstab: static file system information.
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/nvme1n1p2 during curtin installation
UUID=fc939be4-5292-4252-8120-7ef59b177e5b / ext4 defaults 0 1

# /boot/efi was on /dev/nvme0n1p1 during curtin installation
UUID=5187-A8C6 /boot/efi vfat defaults 0 1

# Swap partition
UUID=512d611e-6944-4a57-9748-ea68e9ec3fad	none	swap	sw	0	0

# /dev/mapper/airvideovg2-airvideo /mnt/airvideo ext4 rw,noatime 0 0
UUID=9e78425c-c1f3-4285-9fa1-96cac9114c55 /mnt/airvideo ext4 rw,noatime 0 0

Noticed that I also added the LVM logical volume for /mnt/airvideo, which is my RAID-1 array. The UUID can be obtained by the blkid command. Below is a sample output:

% blkid
/dev/sdf1: UUID="60024298-9915-3ad8-ae6c-ed7adc98ee62" UUID_SUB="fe08d23c-8e11-e02b-63f9-1bb806046db7" LABEL="avs:4" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="552bdff7-182f-40f0-a378-844fdb549f07"
/dev/nvme0n1p1: UUID="r2rLMD-BEnc-wcza-yvro-chkB-1vB6-6Jtzgz" TYPE="LVM2_member" PARTLABEL="primary" PARTUUID="6c85af69-19a0-4720-9588-808bc0d818f7"
/dev/sdd1: UUID="34c6a19f-98ea-0188-bb3f-a5f5c3be238d" UUID_SUB="4174d106-cae4-d934-3ed4-5057531acb3c" LABEL="avs:3" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="2fc4e9ad-be4b-48aa-8115-f32472e61005"
/dev/sdb1: UUID="ac438ac6-344a-656b-387f-017036b0fafa" UUID_SUB="0924dc67-cd3f-dec5-1814-ab46ebdf2fbe" LABEL="avs:1" TYPE="linux_raid_member" PARTUUID="29e7cfce-9e7b-4067-a0ca-453b39e0bd3d"
/dev/md4: UUID="gjbtdL-homY-wyRG-rUBw-lFgm-t0vZ-Gi8gSz" TYPE="LVM2_member"
/dev/md2: UUID="0Nky5e-52t6-b1uZ-GAIl-4Ior-XWTz-wFpHh1" TYPE="LVM2_member"
/dev/sdi1: UUID="5b483ac2-5b7f-4951-84b2-08adc602f705" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="data" PARTUUID="e0515517-9fbb-4d8a-88ad-674622f20e00"
/dev/sdg1: UUID="3d1afb64-8785-74e6-f9be-b68600eebdd5" UUID_SUB="c146cd05-8ee8-5804-b921-6d87cdd4a092" LABEL="avs:2" TYPE="linux_raid_member" PARTLABEL="lvm" PARTUUID="2f25ec17-83c4-4c0b-8653-600283d58109"
/dev/sde1: UUID="34c6a19f-98ea-0188-bb3f-a5f5c3be238d" UUID_SUB="8aabfe5b-af16-6e07-17c2-3f3ceb1514e3" LABEL="avs:3" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="2fc4e9ad-be4b-48aa-8115-f32472e61005"
/dev/sdc1: UUID="ac438ac6-344a-656b-387f-017036b0fafa" UUID_SUB="c188f680-01a8-d5b2-f8bc-9f1cc1fc3598" LABEL="avs:1" TYPE="linux_raid_member" PARTUUID="29e7cfce-9e7b-4067-a0ca-453b39e0bd3d"
/dev/nvme1n1p2: UUID="fc939be4-5292-4252-8120-7ef59b177e5b" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="912e805d-fe68-48f8-b845-9bba0e3e8c78"
/dev/nvme1n1p3: UUID="512d611e-6944-4a57-9748-ea68e9ec3fad" TYPE="swap" PARTLABEL="swap" PARTUUID="04ac46ff-74f3-499a-814d-32082f6596d2"
/dev/nvme1n1p1: UUID="5187-A8C6" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="fe91a6b2-9cd3-46af-813a-b053a181af52"
/dev/sda1: UUID="3d1afb64-8785-74e6-f9be-b68600eebdd5" UUID_SUB="87fe80a1-4a79-67f3-273e-949e577dd5ee" LABEL="avs:2" TYPE="linux_raid_member" PARTUUID="c8dce45e-5134-4957-aee9-769fa9d11d1f"
/dev/md3: UUID="XEJI0m-PEmZ-VFiI-o4h0-bnQc-Y3Be-3QHB9n" TYPE="LVM2_member"
/dev/md1: UUID="usz0sA-yO01-tlPL-12j2-2C5r-Ukhc-9RLCaX" TYPE="LVM2_member"
/dev/mapper/airvideovg2-airvideo: UUID="9e78425c-c1f3-4285-9fa1-96cac9114c55" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdh1: UUID="60024298-9915-3ad8-ae6c-ed7adc98ee62" UUID_SUB="a1291844-6587-78b0-fcd1-65bc367068e5" LABEL="avs:4" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="ed0274b9-21dc-49bf-bdda-566b2727ddc2"

Step 4B (Potentially): If the system boots in the “grub>” prompt, then we will have persuade grub to manually boot by providing the following at the prompt:

grub> set root=(hd9,gpt2)
grub> linux /boot/vmlinuz root=/dev/nvme1n1p2
grub> initrd /boot/initrd.img
grub> boot

To find the root value on the first line, you have use the ls command which is explained in this article. The root parameter on the linux line references the partition which the root directory is mounted. In my case, it was /dev/nvme1n1p2.

After I rebooted, I reinstalled GRUB with the following as super user:

grub-install /dev/nvme1n1

It may also be required to update our initramfs using:

update-initramfs -c -k all

Step 5: At this point the system should reboot and all of the old server’s content should now be on the old hardware. Unfortunately we will need to fix the network interface.

First obtain the MAC address of the network interface using:

% sudo lshw -C network | grep serial   
    serial: 04:42:1a:05:d3:c4

And then we will have to edit the /etc/netplan/00-installer-config.yaml file.

% cat /etc/netplan/00-installer-config.yaml 
# This is the network config written by 'subiquity'
      dhcp4: true
        macaddress: 04:42:1a:05:d3:c4
      set-name: enp6s0
  version: 2

Ensuring the MAC address matches from lshw and that the name is the same as the old system. The name in this example is enp6s0. We then need to execute the following commands to generate the interface.

netplan generate
netplan apply

We need to ensure the name matches because many services on the server have configurations that references the interface name, such as:

  • Configurations in /var/network/interfaces
  • Samba (SMB) (/etc/samba/smb.conf)
  • Pihole (/etc/pihole/setupVars.conf)
  • Homebridge (/var/lib/homebridge/config.json)

Step 6: Fix the router provisioning DHCP IP addresses so that the new server has the same fixed IP address as the old server. This is important because there may be firewall rules referencing this IP address directly. The hostname should have been automatically restored when we restored the partition in Step 3.

Step 7: Our final step is to test the various services and ensure they are working properly. These include:

  • Mail
  • Our web site lufamily.ca
  • Homebridge
  • Plex
  • Pihole (DNS server)
  • SMB (File sharing)

Finally the new system is completed!

New system all up and running!

Another Car Reservation

Last year on July 27, 2021, I placed a reservation for a RAV4 Prime with a Toyota dealer. It is now over one year, and the latest news from the dealer is that I am in position number three. However, they are currently only getting one or two cars a year!

Our initial take on the RAV4 Prime is that it is a hybrid, so it eliminates any range anxiety while still satisfies any day-to-day trips with a 60km all battery range. We have good experience with our Prius Prime which offers us a similar hybrid experience but with only a 35-40 km battery range.

Toyota RAV 4 Prime

While we continue the wait, it looks like many more electric vehicles (other than Tesla’s) are coming on the scene. There are recent additions from BYD, Polestar, Ford, Hyundai, and KIA. What caught my eye from a recent YouTube-surfing-session is the Hyundai Ioniq 5. This vehicle was introduced last December and is now “available for sale” in Canada. It has comparable range (~400km) and charge speed (350 kW DC) with the Tesla Model Y, sans the hefty price tag.


The styling and look of the Ioniq 5 was not appealing to my taste. I then learned that the KIA EV6 is essentially the same vehicle but has a more traditional and sporty styling. Also a quick online build & price investigation showed that the Ioniq is a couple of thousand more expensive if we want to match the AWD long range trims.

So after much YouTube and online research, today I placed another car reservation for the KIA EV6. I opted for the trim named, AWD Long Range with GT-Line Package 1. I skipped the sunroof and the more fancied seats.

The bad news is that the sales guy is projecting a “three years” wait! He says that much of this will depend on supply chain issues. There is a good chance that it will be much sooner than the current projection.

On a side note, here is something else I discovered relating to KIA quality.

JD Power: Click for orignal source

I did not realize KIA ranked so high. The Buick and Dodge brand frankly surprised me as well. I wonder the accuracy on the above report, so take it for what it’s worth.

Nevertheless, I am keeping the RAV 4 reservation to see what options I have in 2023. Today, I also discovered that the Model Y may get a price cut and start sporting the new LFP batteries from CATL.

We will see! Who knew that buying an EV in 2022 is so difficult! This does not bold well for the planet.