Experimental Machine for AI

The advent of the Large Language Model (LLM) is in full swing within the tech community since the debut of ChatGPT by openAI. Platforms such as Google Colab, and similar variants from Amazon and Facebook allows software developer to experiment with LLM’s. The hosted model of the data center based GPU’s makes training and refinement of LLM’s tolerable.

What about using LLM on a local computer away from the cloud?

Projects such as llama.cpp by Georgi Gerganov makes it possible to run the Facebook open sourced Llama 2 model on a single MacBook. The existence of llama.cpp gives hope on creating a desktop that is powerful enough to some local development with LLM’s away from the cloud. This post documents an experimental procedure in building a desktop machine using parts readily available from the Internet to see if we can do some AI development with LLM’s.

Below is a list of sourced parts from EBay, Amazon and CanadaComputers, a local computer store. All prices are in Canadian dollars and includes relevant taxes.

NVIDIA Tesla P40 24GB GDDR5 Graphics Card (sourced from EBay)$275.70
Lian-Li Case O11D Mini -X Mid Tower Black (sourced from Amazon)$168.49
GDSTIME 7530 75mm x 30mm 7cm 3in 12V DC Brushless Small Mini Blower Cooling Fan for Projector, Sleeve Bearing 2PIN (sourced from Amazon)$16.94
CORSAIR Vengeance LPX 64GB (4 x 32GB) DDR4 3200 (PC4-25600) C16 1.35V Desktop Memory – Black (sourced from Amazon)$350.28
AMD Ryzen 7 5700G 8-Core, 16-Thread Unlocked Desktop Processor with Radeon Graphics (sourced from Amazon)$281.35
Noctua NH-D15 chromax.Black, Dual-Tower CPU Cooler (140mm, Black) (sourced from Amazon)$158.14
Asus AM4 TUF Gaming X570-Plus (Wi-Fi) ATX motherboard with PCIe 4.0, dual M.2, 12+2 with Dr. MOS power stage, HDMI, DP, SATA 6Gb/s, USB 3.2 Gen 2 and Aura Sync RGB lighting (sourced from Amazon)$305.09
Samsung 970 EVO Plus 2TB NVMe M.2 Internal SSD (MZ-V7S2T0B/AM) (sourced from Amazon)$217.72
Lian Li PS SP850 850W APFC 80+ GOLD Full modular SFX Power Supply, Black (sourced from CanadaComputers)$225.99
Miscellaneous 120mm case fans and cables purchased from CanadaComputers$63.17

The total cost of the above materials is $2,062.87 CAD.

The Nvidia Tesla P40 (Pascal Architecture) specializes for Inferencing limited to INT8 based operations and does not support any FP related operations, so it may not be optimal for machine learning. However recent claims have been made that INT8 / Q8_0 quantization can yield some promising results. Let us see what our experimentation will yield once the machine is built.

A custom design 3D fan shroud has to be designed and 3D printed because the P40 does not natively come with active cooling. The P40 is originally designed to operate in a data center so cooling is provided by the server chassis. The custom shroud design is posted on Thingiverse and some photos of the finished shroud is shown below.

Note that M3 screws were used to secure the shroud to the P40 GPU card. The GDSTIME fan came with the screws.

I also made a mistake by initially getting a 1000W ATX power supply that ended not fitting the case, because the case is built for SFX and SFX-L power supplies. Lesson learned!

Once the machine is built I performed a 12 hours MemTest86+. It turned out that running the memory at the XMP profile was a bit unstable. I had to clock the memory back from its 3200MHz rating to 3000MHz.

After more than 12 hours with 3 passes.

The BIOS settings had to be configured so that Resize BAR is ON. This is required for the P40 to function properly.

Turn on Resize BAR

The next step is to install Ubuntu 22.04.3 LTS with Nvidia GPU and CUDA drivers. The latter was quite challenging. The traditional way of installing using the package manager did not work. The best way is to goto this site, and pick the run file like below:

Beside to use the runfile

The run file had to be run in recovery mode using the console because the installation will fail if an X11 window manager is running. Also all previous Nvidia drivers had to be removed and purged. The Ubuntu default installation process may have installed them.

A detail that was left out of the instructions is to set the appropriate shell paths once the installation is completed. The following changes were made with /etc/profile.d so that all users can benefit. If the login shell is using zsh, then /etc/zsh/zshenv has to be changed. Without this change, commands such as nvcc and other CUDA toolkit commands will not be found. The same is true for CUDA related share libraries.

$cat /etc/profile.d/cuda-path.sh

export CUDA_HOME="/usr/local/cuda"

if [[ ! ${PATH} =~ .*cuda/bin.* ]]
    export PATH="${PATH}:/usr/local/cuda/bin"

if [[ ! ${LD_LIBRARY_PATH} =~ .*cuda/lib64.* ]]
    export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/cuda/lib64"

if [[ ! ${LD_LIBRARY_PATH} =~ .*/usr/local/lib.* ]]
    export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib"

In this hardware configuration the AMD CPU has integrated graphics, and the P40 does not have any HDMI or DisplayPort connections. We need to change the X11 configuration so that it will only use the AMD CPU while dedicating the P40 GPU for CUDA based computation. The following configurations have to be made in /etc/X11/xorg.conf:

$cat /etc/X11/xorg.conf

Section "Device"
    Identifier      "AMD"
    Driver          "amdgpu"
    BusId           "PCI:10:0:0"

Section "Screen"
    Identifier      "AMD"
    Device          "AMD"

The BusId can be obtained using the lspci command and be sure to change any hexadecimal notations to decimal in the configuration file. Without this xorg.conf configuration, the Ubuntu desktop will not start properly.

When everything is done properly, the command nvidia-smi should show the following:

Fri Aug 25 17:33:31 2023
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|   0  Tesla P40                      Off | 00000000:01:00.0 Off |                  Off |
| N/A   22C    P8               9W / 250W |      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |

| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|  No running processes found                                                           |

The machine is now ready for user account configurations.

A quick video encoding using ffmpeg with hardware acceleration and CUDA was performed to test the GPU usage. It was a bit of a challenge to compile ffmpeg with CUDA support. This is when I found out that I was missing the PATH configurations made above.

For good measure, gpu-burn was run for an hour to ensure that the GPU is functioning correctly.

Next step is to download and setup the tool chain for LLM development. We will save that for another posting.

New 1.5Gbps Internet Service

On April, 4th, I received a promotional offer from Rogers offering Ignite Internet service at 1.5Gbps plus Streaming for $114.99 per month.

I procrastinated a bit because I wanted to make sure that I can actually make use of this service. However, when I checked my bill for April, I noticed that my total monthly charges is at $102.99.

Note the above price prior to discount is at $117.99. I was curious to see if Rogers can get me a good deal without the Streaming service. I called Rogers support line and received a person who was not very helpful and simply just quoted conditions and deals to me. AI will do a number of these types of people soon.

I decided to try an alternative route by using Twitter (@RogersHelp). I direct messaged Rogers on Twitter and received wonderful help. They offered me the 1.5Gbps service at only $104.99 (with a 24 months commitment). This will be somewhat on par with my current payment and I will get 50% more throughput.

There is another question. Will my networking equipment make use of the 1.5Gbps? My networking setup has the Rogers Ignite WiFi Gateway (ARRIS Group, XB7 Modem) and is connected with a Cat5e cable to my Unifi Dream Machine Pro, using one of its 1Gbps RJ45 port.

Rogers XB7 Modem Unifi Dream Machine (UDM) Pro(Firewall / Router)
Home Network
Home Network
Text is not SVG – cannot display

How can we overcome the 1Gbps limit on the UDM Pro’s RJ45 port? Luckily the UDM Pro has an 10G SFP+ port as well. I went to Amazon and purchased a 10G SFP+ RJ45 Copper Module Transceiver.

The above will auto negotiate a 2.5Gbps to 10Gbps connection from the XB7 to my UDM Pro. Of course I will not be getting 2.5G or 10G speeds. These are just the physical maximum per the respective devices. Rogers will throttle my inbound and outbound traffic to 1.5Gbps and 50Mbps respectively.

After installing the SFP+ module and rewire the existing Cat5e cable, I had to reboot the both the XB7 modem and the UDM Pro. Once everything came back up, I had another problem. How do I test that I actually get 1.5Gbps? I cannot do it from any WiFi devices or any wired devices in my house because they are all limited to the 1Gbps port speed from my networking switches. Once again, Unifi had thought of this already and provide a speed test functionality on its management dashboard.

The tested speed seems to be better than expected.

So you can see from the above screen shot, that we are now getting what we are paying for. I also performed a double test from two different machines that are routed to a switch that has a 10Gbps connection to my UDM Pro, and each machine received a 700Mbps to 800Mbps download speed, which is around 1.5Gbps in aggregate. Mission accomplished.

Unifi just came out with a new firmware update that enable the UDM Pro to perform load balancing of more than one WAN connection. When the SkyLink service becomes more economically feasible, we can attach a satellite based internet service as a compliment to the existing Rogers service. This way during a power outage, we can continue to get Internet.

Playing with Proxmox

Prior to the holidays in 2022, I upgraded my media NAS server as detailed here. After this upgrade, I repurposed the old server’s components and built another PC.

Originally I was going to use this extra PC as a simple online media encoder, since encoding videos in the HEVC codec takes a lot of CPU power. I did this for about a month. My son, Kalen had an old GTX1060 6GB graphics card that he was going to place on Kijiji for resale. I offered to purchase this graphics card off of him so that that I can pair it up with this repurposed PC. The new idea was to turn this PC into my gaming PC. I don’t do many 3D intensive gaming, so an older GPU is certainly good enough for me.

Off I went installing Windows 10 Pro on the PC. I also discovered at this time the Windows Subsystem for Linux (WSL). I thought it would be a wonderful idea for me to have the gaming PC and not lose the ability for the PC to double as a media encoder through the use of a Linux distribution using WSL. My hope is that Linux with WSL will yield near metal based performance. Long story short, the performance of ffmpeg, the tool that I use for video encoding, was disappointing. Apparently there is a bug in WSL v2 that forced ffmpeg to only use 50% of its CPU power. There was nothing wrong with the concept of having a dual purpose PC for gaming and a handy Linux distribution for other endeavours.

The problem is with the Windows hosted Hypervisor, a software layer that usually runs between the hardware and the operating. I know of another hypervisor called Proxmox. This is a perfect opportunity for me to try Proxmox out. Before I installed Proxmox, I maxed out the memory of this repurposed PC to 64GB. It only had 16GB before and I thought this would not be enough.

One of the worries I had was how to get the raw GPU performance from Proxmox? Apparently there is a GPU passthrough option. Before installing Proxmox, I had to make some BIOS adjustments on the PC.

  • Enable IOMMU
  • Enable SVM Mode (same as Intel VT-x)
  • Enable AMD ACS

Only the SVM Mode is required for Proxmox, the other two are required for GPU Passthrough. After I installed the Proxmox server, I followed the instructions outlined in the following sites:

  1. From 3os.org: GPU Passthrough to VM;
  2. From pve.proxmox.com;
  3. And from reddit.

The first site was more clear and was the most helpful. I used the second and third sites as an alternate source and backup reference. Thanks to the above sites, I was able to get Proxmox running and created two virtual machines (VM’s). The first is an Ubuntu distribution called workervm and the second is a Windows 10 Pro instance with a GPU passthrough, called win10. Below is a screenshot of the Proxmox control administration site.

Proxmox control panel (click to enlarge)

Below is the workervm (Linux VM) configuration:

workervm configuration for Ubuntu instance

I had to make sure the processor type is set to [host] to get the most performance out of the virtual CPU’s. The Windows VM configuration uses a different BIOS, specifically a UEFI BIOS. We also have to ensure that the Machine type is set to q35. The Windows VM also has the EFI Disk and TPM State configured, and of course the extra PCI Device to represent our GPU passthrough card. Check out the full configuration for the Windows 10 VM below:

win10 configuration for Windows 10 Pro instance

After installing Windows 10 Pro, the network interface is not recognized. To remedy this situation I had to install virtio-win as described by this site here. After the installation of virtio-win, and a reboot. I had networking, connectivity to the Internet, and the Device Manager output from the Windows 10 Pro instance as shown below. Notice that Windows recognized the native NVIDIA GeForce GTX 1060 6GB card.

Windows 10 Pro VM instance Device Manager

I tried to test out the GPU throughput with some 3D rendering demos and tested a couple of games from Steam using Remote Desktop. The performance was okay, but not stellar, and could have been better. I did some more research, and apparently Parsec, a virtual desktop sharing tool, is probably better for remote gaming.

I went ahead and installed Parsec on both the Windows 10 Pro VM, and on my Mac mini, which I used to remotely play games on the previous VM. This worked out to be quite well.

Now the repurpose PC is a Proxmox server hosting as many VM’s that the hardware can bear. The workervm instance can be used for video encoding and other generic Linux oriented work or trials. The win10 instance will be used for gaming and hosting our tax filing software, called TurboTax, which only runs on Windows.

In the near future, I will also be testing out Proxmox with virtual containers instead of machines. The containers are more light weight and less resource intensive. It will be another new adventure here.

Panel Snow Coverage

Today is January 13, 2023. We had an icy snow storm last night that lasted until this morning, and I was curious what the roof condition was like. Just how much of the panels were covered in snow?

Solar energy for today

Our peak energy production was at around 11am when we generated a little over 800Wh, which is inlined with what we kind of get on a cloudy, misty, winter day. In contrast, the best we got so far was on January 7th at 1pm. We generated 5,494Wh. That was a sunny day with no snow coverage on the panels.

A quick drone survey of our roof this afternoon at around 3pm.

I was kind of impressed that we got that much with so much of the panels covered. Watch the above video to see just how much of the panels are covered today. Our total production for today is only about 3,400Wh.

Below are the stats per panel.

Per panel generation statistics for today.

As you can see above, every panel contributed even the covered ones! There will be two sunny days over the weekend, so we will see!

Update: 2023-01-14

I did another roof survey with my drone, seeing that today it was a sunny sky day.

Roof survey on Jan. 14 (day after storm)
Solar energy production on Jan. 14

We have generated over 10,000 Wh of energy today about 3 times more than yesterday. The survey was conducted when it was still -6 ºC outside, so way below freezing.

Managing Audio Books with Plex

Library Setup

I have a membership with Audible and I sometimes also get other audio book sources. Recently I experimented with combining all of my audio books into a centralized place. Since I already have a Plex server running, I thought it would be a good place to do this.

I did a little research and came across a couple of very helpful articles:

  • A Reddit article;
  • and some really detail information on GitHub;

The main points are:

  • I have a single folder to store all of my audio books. Inside the folder, each audio book is stored as an “m4b” file.
  • Ensure that audio books have a poster image and that its artist and album_artist tags are set to the author. Where appropriate, the audio book should also contain chapter metadata.
  • Download and install the Audnexus agent;
  • Create a music library on Plex by adding the audio book folder, and set the agent to Audnexus
Note the Agent setting
  • Ensure that the advanced option of “Store track progress” is checked.
Ensure that Store Track Progress is checked!

Each book in the library will be represented as an album, and the author will be mapped to album artist. Once the library is created, you can download and play the audio books from the desktop using the Plex app. However, the more common use case is to listen to the audio books while on the go.

Using Prologue to Play Audio Books

We first have to download the Prologue App. I did not get any of the In-App premium functionality, and just stayed with the free version.

Point the app to my Plex server’s URL, and all the audio books from the library should now be accessible and playable on the iPhone or iPad with chapter, bookmark, and last-left-position support.

This is a really neat solution, and I am impressed how Plex and Prologue together formed a dynamic duo in this manner.

A Weekend Programming Challenge

My wife and I were watching  Lighter & Princess (点燃我,温暖你). In episode 5, a smart programmer completed a heart animation as part of a programming test in a university course. Everyone of his classmates had a mediocre and simplistic implementation.

My wife challenged me to do the same. I could have done this with SVG and Javascript, but I took the opportunity to learn more Python and Qt. The latter I have not used at all and was always curious how well it worked with Python. I decided to use the PyQt package instead of the PySide package.

I hacked this together today resulting in this rendition:

Click above to see it in action

The PyQt implementation worked really well across Windows, Mac, and Ubuntu Linux. I suppose the Javascript and SVG would have done the same, but I learned something new in this case.

The source code and the installation instructions are at:

Click above to goto GitHub

Media Server Upgrade 2022 (Part 2)

Part 1

In the first part of this post, I talked about making sure all the new hardware that I recently purchased works. Yesterday, upgrading from Ubuntu 20.04 LTS to 22.04 LTS was super simple. Unfortunately, that was the end of the easy part.

I thought I could just image by old boot drive and make a carbon copy of it on my new boot drive. My old boot drive is a simple SATA 512GB SSD, and my new boot drive is an NVMe M.2 1TB SSD plugged directly to the motherboard. The copying was pretty simple, but because the drives differ in size, I had to relayout the partition table with the new drive once the copy is completed. I did this with the parted command.

Unfortunately the new boot drive did not want to boot. At this point I had to do some research. The most helpful articles were:

Both of the above articles were an excellent refresher on how GRUB works. I have used GRUB since the beginning, but one gets super rusty when these types of tasks are only performed once every three or six years!

Instead of detailing what went wrong, I will just explain what I should have done. This way if I need it again in the future, it is here for my reference.

Step 1: Perform a backup of the old boot drive from a Live USB in shell mode. This is done on my server on a nightly basis. This method is clearly described on the Ubuntu Community Help Wiki.

Following this method I will end up with a compressed tar archive for my entire root directory, skipping some runtime and other unwanted directories.

Step 2: After installing a fresh install of the new Ubuntu LTS Server operating system on the new server and boot drive, I proceeded to backup the new boot with the same technique used in Step 1. I stored the backup of the new install on another external SSD drive that I have lying around. Also it is important that new boot drive partition layout of the new install contains a swap partition.

Step 3: I then restore the most recent backup (done in Step 1) of the old boot drive to the new boot drive. I then replaced the /boot/grub directory with the new contents from the new install which was backed up in Step 2. The new GRUB is already installed when we performed a brand new installation on the drive. We just want to make sure the boot partition matches the /boot/grub contents.

Step 4: We also need to fix up the /etc/fstab file because it contains references to drive devices from the old hardware. Paid special attention the main data partition and the swap partition. It should look something like this:

# /etc/fstab: static file system information.
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/nvme1n1p2 during curtin installation
UUID=fc939be4-5292-4252-8120-7ef59b177e5b / ext4 defaults 0 1

# /boot/efi was on /dev/nvme0n1p1 during curtin installation
UUID=5187-A8C6 /boot/efi vfat defaults 0 1

# Swap partition
UUID=512d611e-6944-4a57-9748-ea68e9ec3fad	none	swap	sw	0	0

# /dev/mapper/airvideovg2-airvideo /mnt/airvideo ext4 rw,noatime 0 0
UUID=9e78425c-c1f3-4285-9fa1-96cac9114c55 /mnt/airvideo ext4 rw,noatime 0 0

Noticed that I also added the LVM logical volume for /mnt/airvideo, which is my RAID-1 array. The UUID can be obtained by the blkid command. Below is a sample output:

% blkid
/dev/sdf1: UUID="60024298-9915-3ad8-ae6c-ed7adc98ee62" UUID_SUB="fe08d23c-8e11-e02b-63f9-1bb806046db7" LABEL="avs:4" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="552bdff7-182f-40f0-a378-844fdb549f07"
/dev/nvme0n1p1: UUID="r2rLMD-BEnc-wcza-yvro-chkB-1vB6-6Jtzgz" TYPE="LVM2_member" PARTLABEL="primary" PARTUUID="6c85af69-19a0-4720-9588-808bc0d818f7"
/dev/sdd1: UUID="34c6a19f-98ea-0188-bb3f-a5f5c3be238d" UUID_SUB="4174d106-cae4-d934-3ed4-5057531acb3c" LABEL="avs:3" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="2fc4e9ad-be4b-48aa-8115-f32472e61005"
/dev/sdb1: UUID="ac438ac6-344a-656b-387f-017036b0fafa" UUID_SUB="0924dc67-cd3f-dec5-1814-ab46ebdf2fbe" LABEL="avs:1" TYPE="linux_raid_member" PARTUUID="29e7cfce-9e7b-4067-a0ca-453b39e0bd3d"
/dev/md4: UUID="gjbtdL-homY-wyRG-rUBw-lFgm-t0vZ-Gi8gSz" TYPE="LVM2_member"
/dev/md2: UUID="0Nky5e-52t6-b1uZ-GAIl-4Ior-XWTz-wFpHh1" TYPE="LVM2_member"
/dev/sdi1: UUID="5b483ac2-5b7f-4951-84b2-08adc602f705" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="data" PARTUUID="e0515517-9fbb-4d8a-88ad-674622f20e00"
/dev/sdg1: UUID="3d1afb64-8785-74e6-f9be-b68600eebdd5" UUID_SUB="c146cd05-8ee8-5804-b921-6d87cdd4a092" LABEL="avs:2" TYPE="linux_raid_member" PARTLABEL="lvm" PARTUUID="2f25ec17-83c4-4c0b-8653-600283d58109"
/dev/sde1: UUID="34c6a19f-98ea-0188-bb3f-a5f5c3be238d" UUID_SUB="8aabfe5b-af16-6e07-17c2-3f3ceb1514e3" LABEL="avs:3" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="2fc4e9ad-be4b-48aa-8115-f32472e61005"
/dev/sdc1: UUID="ac438ac6-344a-656b-387f-017036b0fafa" UUID_SUB="c188f680-01a8-d5b2-f8bc-9f1cc1fc3598" LABEL="avs:1" TYPE="linux_raid_member" PARTUUID="29e7cfce-9e7b-4067-a0ca-453b39e0bd3d"
/dev/nvme1n1p2: UUID="fc939be4-5292-4252-8120-7ef59b177e5b" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="912e805d-fe68-48f8-b845-9bba0e3e8c78"
/dev/nvme1n1p3: UUID="512d611e-6944-4a57-9748-ea68e9ec3fad" TYPE="swap" PARTLABEL="swap" PARTUUID="04ac46ff-74f3-499a-814d-32082f6596d2"
/dev/nvme1n1p1: UUID="5187-A8C6" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="fe91a6b2-9cd3-46af-813a-b053a181af52"
/dev/sda1: UUID="3d1afb64-8785-74e6-f9be-b68600eebdd5" UUID_SUB="87fe80a1-4a79-67f3-273e-949e577dd5ee" LABEL="avs:2" TYPE="linux_raid_member" PARTUUID="c8dce45e-5134-4957-aee9-769fa9d11d1f"
/dev/md3: UUID="XEJI0m-PEmZ-VFiI-o4h0-bnQc-Y3Be-3QHB9n" TYPE="LVM2_member"
/dev/md1: UUID="usz0sA-yO01-tlPL-12j2-2C5r-Ukhc-9RLCaX" TYPE="LVM2_member"
/dev/mapper/airvideovg2-airvideo: UUID="9e78425c-c1f3-4285-9fa1-96cac9114c55" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdh1: UUID="60024298-9915-3ad8-ae6c-ed7adc98ee62" UUID_SUB="a1291844-6587-78b0-fcd1-65bc367068e5" LABEL="avs:4" TYPE="linux_raid_member" PARTLABEL="primary" PARTUUID="ed0274b9-21dc-49bf-bdda-566b2727ddc2"

Step 4B (Potentially): If the system boots in the “grub>” prompt, then we will have persuade grub to manually boot by providing the following at the prompt:

grub> set root=(hd9,gpt2)
grub> linux /boot/vmlinuz root=/dev/nvme1n1p2
grub> initrd /boot/initrd.img
grub> boot

To find the root value on the first line, you have use the ls command which is explained in this article. The root parameter on the linux line references the partition which the root directory is mounted. In my case, it was /dev/nvme1n1p2.

After I rebooted, I reinstalled GRUB with the following as super user:

grub-install /dev/nvme1n1

It may also be required to update our initramfs using:

update-initramfs -c -k all

Step 5: At this point the system should reboot and all of the old server’s content should now be on the old hardware. Unfortunately we will need to fix the network interface.

First obtain the MAC address of the network interface using:

% sudo lshw -C network | grep serial   
    serial: 04:42:1a:05:d3:c4

And then we will have to edit the /etc/netplan/00-installer-config.yaml file.

% cat /etc/netplan/00-installer-config.yaml 
# This is the network config written by 'subiquity'
      dhcp4: true
        macaddress: 04:42:1a:05:d3:c4
      set-name: enp6s0
  version: 2

Ensuring the MAC address matches from lshw and that the name is the same as the old system. The name in this example is enp6s0. We then need to execute the following commands to generate the interface.

netplan generate
netplan apply

We need to ensure the name matches because many services on the server have configurations that references the interface name, such as:

  • Configurations in /var/network/interfaces
  • Samba (SMB) (/etc/samba/smb.conf)
  • Pihole (/etc/pihole/setupVars.conf)
  • Homebridge (/var/lib/homebridge/config.json)

Step 6: Fix the router provisioning DHCP IP addresses so that the new server has the same fixed IP address as the old server. This is important because there may be firewall rules referencing this IP address directly. The hostname should have been automatically restored when we restored the partition in Step 3.

Step 7: Our final step is to test the various services and ensure they are working properly. These include:

  • Mail
  • Our web site lufamily.ca
  • Homebridge
  • Plex
  • Pihole (DNS server)
  • SMB (File sharing)

Finally the new system is completed!

New system all up and running!

Media Server Upgrade 2022

On May 15th, 2019 (more than three years ago), I performed a performance boost to my media server by upgrading its CPU, Motherboard, and Memory. You can read that experience in this post.

Today, I am going to be doing the same. It looks like we are on a cadence of every 3 years or so to do a spec bump. This time around we are also changing the same items, but will include the power supply as well in the swap. I also decided to swap the boot drive hardware from an old SSD drive to an NVME drive. All of this resulted in the following hardware acquisitions, all from Amazon, which I find them to have lower pricing (when factoring free shipping through Prime), than Newegg even during Black Friday and Cyber Monday offers.

  • AMD Ryzen 7 5700G 8-Core, 16-Thread Unlocked Desktop Processor with Radeon Graphics
  • ASUS TUF GAMING B550-PLUS AMD AM4 (3rd Gen Ryzen™)
  • G.SKILL Ripjaws V Series DDR4 3600MHz 32GB(16GBx2) Memory Kit
  • ASUS ROG Strix 850W Gold PSU
  • Samsung 980 PRO SSD 1TB – M.2 NVMe

The above totalled $1045.60 CAD.

The plan is to spend the time today to roughly test out all the new hardware.

Test Setup

I quickly did a skeleton setup to make sure Ubuntu 22.04.01 Server Edition works with all hardware involved, especially the networking.

Memory Test

Once I know Ubuntu server is working good, I am now testing the server’s new 32GB DDR4 memory. This is running as I write this post and will let it run overnight.

The plan for tomorrow is to upgrade the current media server from Ubuntu 20.04.5 LTS to Ubuntu 22.04.1 LTS. Once this is done, I can then backup everything, and move the new hardware into the old casing and hope everything works.

Part 2

XWPro Configurations

This is going to be a fairly technical post on the topic of my Conext XWPro battery inverter configurations. I am writing this post primarily to document my experience and my current rationale, and for my future, forgetful self.

Previously I had my Grid Support SOC (State of Charge) and the Recharge SOC set to 40%. With these settings, the battery will be used (anytime during the day and night) until it discharged to 40%, which will initiate a charge cycle that will charge the battery back to 100%. Under normal circumstances, the battery will typically discharge during a very cloudy day, but mostly in the evenings and at night.

I had the above settings because I stupidly thought I should stay off the grid as much as possible. The intent is to try to charge the batteries in the evening during off peak hours, and not to use the grid at all during on peak hours. These settings certainly accomplish this, but at the expense of shortening battery life. Another big downside with this approach is that charging the battery through the Conext XWPro inverters only achieves around 83% efficiency. This observation is based on my real-time data observations from the actual inverters.

Yesterday I noticed that my batteries are reporting a State of Health (SOH) drop to 99% instead of 100%. This was a bit alarming given only 6 months of uses. I also realized from the Alectra invoices that Time of Use (ToU) is not a factor in Alectra’s billing calculations. All of this resulted in a shift in my thinking. We will now use the grid as our primary battery, and preserve our Lithium Ion as our backup batteries only. Time shifting of loads will no longer be my primary concern since it is no longer worth it with zero benefits.

To do this, I have set the Grid Support SOC to 90% and the Recharge SOC set to 85%. This way immediately after a charge cycle, the battery will be used a little bit to draw down from 100% to 90%. This has two benefits in my opinion. The first is to get some charge flow through the batteries, so it is not only sitting there. The second is that it leaves a 10% SOC gap. If we have a power failure during a sunny day, there is space for the excess solar production to go without tripping the solar inverters.

The 5% gap between Grid Support and Recharge is currently a guess. My thinking is that over time the charge on the Lithium batteries will leak and it will trigger a recharge cycle. Of course I did not want to set the Recharge the same as the Grid Support, because this will cause a constant recharge loop which defeats the purpose of preserving battery life. I do not know how long it will take to naturally draw down from 90% to 85%. This is why it is still a guess at this point. If there is no leakage, which is great news because it shows how good the batteries are, then I will have to trigger a recharge cycle at least once a month just to keep the charge flow within the battery’s chemistry.

For now I will live with the new settings and see how often the battery cycles. If it only cycles once every one or two months, then that is perfect. If it does not cycle through more than three months, then I may have to add the monthly charging cycle logic into my custom controller.

Reading Our Net Smart Meter, Carbon Neutrality?

We now have been running our net smart meter for more than a day now. I mentioned that we got our new net meter on this previous post.

Of course I am now curious how to read the meter so that I can decipher how much electricity we sent back to the grid. Here is a short video of what the meter is showing:

Meter Display Sequence

Initially the displayed information is quite cryptic, but looking at the meter’s label, I found this group of small prints.

These labels essentially tells us what is going on. The LED display cycles through 5 modes in total. The initial display is a segment test, which means all segments of the LED are displayed. This is a simple test to ensure that the LED display itself is functioning correctly. Next, it shows LST003, indicating that the next number it shows will be the amount of kWh of electricity that we ended up consuming or using. This is followed by LST004, another label indicating that the following number is the amount of kWh of electricity that is sent back to the grid.

Now with this new found knowledge, the above video shows that we used 13 kWh and exported 103 kWh since the meter was installed in the afternoon of May 4th.

In about 1.5 days, and bright sunny day yesterday, we generated and provided to our community electricity grid with a net of 90 kWh of energy.

Excerpt from the CBC article from Oct. 8th, 2021

There are 4 people in our house right now, and according to a recent CBC article our average carbon footprint is about 14.2 tonnes of CO2 per person. Doing a little more research, I found this white paper titled, “A Clearer View on Ontarios Emissions June 2019“. On page 8 of this paper, we see an annual average emissions factor (AEF) of 31 grams of CO2 per kWh. One tonne is 1,000,000 (a million) grams. This means to offset one individual, we need to offset 14,200,000 grams of CO2, and using the AEF this is equivalent to approximately 458,065 kWh!

To put this big number in perspective, I think our last month’s electricity bill only shows us using around 1,200 kWh of electricity.

It is clear that we will not be able to offset one of us, never mind all four of us by just using solar ourselves (at least not in Ontario). The idea of carbon neutrality is still a long ways off, and the above numbers show that we cannot do it alone. It will require every industry to do its part.

Update 2022-05-26 2:45pm: Took another reading outside. Used 313 kWh, Exported 1018 kWh, a net of 705 kWh. This with about 23 days of operation since May 04th.