Kang's Thoughts

NVMe SSD with LVM Cache

I have been a huge fan of Apple’s fusion drives. They are an excellent compromise for affordable mass storage while still able to give you SSD performance. The concept is simple pair a fast but small SSD drive with a large but slow and much affordable, mechanical HDD. You get good performance and have lots of storage without breaking the bank.

I have falsely assumed that this capability only existed with Apple’s macOS operating system. This week I was pleasantly surprised to have discovered that LVM Cache can do more or less the same thing on Linux. This new found knowledge along with an excellent deal on a 500GB NVMe Samsung 970 Evo Plus M.2 drive gave me the itch to experiment this weekend with my NAS media server.

The hardware was easy enough to install, but I had to move one of the existing SATA connection because the M.2 slot on the motherboard shared a PCIe bus with a pair of SATA connections. Luckily I bothered to check the motherboard manual, otherwise I would have been scratching my head while the server fail to boot.

The software configurations were a bit more involved. Before I purchased the NVMe card, I did some experimentation with two external USB drives, one SSD and one HDD. I found this article to be super helpful in configuring LVM Cache with my test drives. However, these configurations were not fully restored after a reboot. After many hours of research on the Internet, I found this article indicating that my Ubuntu Linux distribution was missing the thin-provisioning-tools package. I also had experimented between the two different cache modes that were available, writethrough and writeback. I found out that the write back mode was a bit buggy and did not sync the cache and the storage drive. Yet another article to the rescue.

lvchange --cachesettings migration_threshold=16384 vg/cacheLV

I preferred the write back mode due to its better write performance characteristics. Apparently to fix the issue, I have to increase the migration threshold to something larger than the default of 2048 because the chunk size was too large.

Here are the steps that I did to configure my existing logical volume (airvideovg2/airvideo) to be cached by the NVMe drive that I just purchased. I first have to partitioned the NVMe drive.

Model: Samsung SSD 970 EVO Plus 500GB (nvme)  Disk /dev/nvme0n1: 500GB  Sector size (logical/physical): 512B/512B  Partition Table: gpt  Disk Flags:      Number  Start   End    Size   File system  Name     Flags   1      1049kB  500GB  500GB               primary

Create an LVM physical volume with the NVMe partition that was created previously /dev/nvme0n1p1 and add it to the existing airvideovg2 volume group.

sudo pvcreate /dev/nvme0n1p1 

sudo vgextend airvideovg2 /dev/nvme0n1p1

Create a cache pool logical volume and set its cache mode to write back and establish the migration threshold setting.

sudo lvcreate --type cache-pool -l 100%FREE -n lv_cache airvideovg2 /dev/nvme0n1p1
  
sudo lvchange --cachesettings migration_threshold=16384 airvideovg2/lv_cache

sudo lvchange --cachemode writeback airvideovg2/lv_cache

Finally link the cache pool logical volume to our original logical volume.

sudo lvconvert --type cache --cachepool airvideovg2/lv_cache airvideovg2/airvideo

Now my original logical volume is cached and I have gained SSD performance economically on my 20TB RAID setup for less than $200. Below is my final volume listing.

$ sudo lvs -a    LV               VG          Attr       LSize   Pool       Origin           Data%  Meta%  Move Log Cpy%Sync Convert    airvideo         airvideovg2 Cwi-aoC---  20.01t [lv_cache] [airvideo_corig] 0.01   11.78           0.00                [airvideo_corig] airvideovg2 owi-aoC---  20.01t                                                                        [lv_cache]       airvideovg2 Cwi---C--- 465.62g                             0.01   11.78           0.00                [lv_cache_cdata] airvideovg2 Cwi-ao---- 465.62g                                                                        [lv_cache_cmeta] airvideovg2 ewi-ao----  64.00m                                                                        [lvol0_pmspare]  airvideovg2 ewi-------  64.00m

We can also use the command below to get a more detail listing.

sudo lvs -a -o+name,cache_mode,cache_policy,cache_settings,chunk_size,cache_used_blocks,cache_dirty_blocks

Upgrade completed. We’ll see how stable it is in the future.

Media Server Upgrade

Two and half years ago, I performed a CPU and motherboard upgrade to my media server. You can read the account here.

Although the AMD Athlon 5350 APU was energy efficient, it proved to be under power for on demand video encoding when Plex wanted to transcode video for a player on a device that is not compatible with the playing video. For example, when an Apple TV (not 4K) wants to play 4K material from Plex on my media server, the server will have to transcode the 4K material to a compatible 1080p format. Unfortunately, this is very CPU intensive and if more than one person in the house hold is trying to do the same thing, which is not unheard of, this causes stuttered playback issues.

Given the choice between saving a few dollars a year versus usability, I choose usability. Therefore I started to research what I need for the upgrade. My goal is upgrade the system so that transcoding will not be an issue and I can also use the system for future video encoding of security camera footages. We can also use the system for background video encoding of family videos as well.

I continue to prefer the AMD brand, and decided on the following combo:

AMD Ryzen 5 2400G Processor with Radeon RX Vega 11 Graphics (YD2400C5FBBOX)
GIGABYTE B450 AORUS M Motherboard
Corsair Vengeance LPX 16GB (2x8GB) DDR4 DRAM 2666MHz (CMK16GX4M2A2666C16)

The above were all purchased through Amazon and cost me a grand total of $473.24. The AMD CPU was the most expensive part costing almost $190.

Taking out the old motherboard and CPU combo and replacing them with the new parts went smoothly. The side SATA connectors bucked against one of my HDD chassis so I opted not to use them, and decided to connect all of my RAID SATA connectors to the SATA accessory card that I purchased and discussed in this post.

Last time I did an upgrade like this, the Ubuntu operating system had no problems and booted without any issues. Unfortunately, this time is very different. After the machine posted, Ubuntu booted into a blank, black screen. After some research, I learned to reboot the Ubuntu kernel with the nomodeset option. I learned to press and hold the shift key so that I can select the desired kernel that I wanted via the GRUB menu, and I learned to press the ‘e’ key in the GRUB menu to modify the boot options. Finally pressing F10 to boot with the custom changes (effective for only one time).

The above trick got me a login prompt. After I gained access to the command prompt, I noticed that the kernel did not recognize any ethernet devices. I now have a machine that is not connected to the network. After some more Internet research I found out that the current 4.15 Linux Kernel that I have is insufficient to run on the Raven Ridge architecture, the AMD code name for the Zen CPU and Vega GPU combination on a single chip. I have to upgrade to the 4.18 Linux Kernel.

However I cannot upgrade through the Internet, because the machine is not on the Internet. I have to download the Debian packages on a USB stick with another machine and manually install them. At this point, I learned that you cannot simply download a single package for this. I had to decide whether to go with the Linux Mainline Kernel packages or go with the Ubuntu HWE (Hardware Enablement) packages. After reading through Ubuntu’s LTS Enablement Stack article, I decided to HWE packages. I found the linux-generic-hwe packages and their prerequisites on pkgs.org. This took several iterations as I did not get all the dependent packages on the first try.

Once all the packages were installed, the machine booted without the need for the nomodeset option. However, the internet interface device was still not there. I had to run the command netpath, to find out that new motherboard’s ethernet device’s logical name was em1. To register the new logical name, I had to edit /etc/network/interfaces file.

Finally, the machine booted with an active ethernet connection. As a sanity check, I executed:

sudo apt-get install --install-recommends linux-generic-hwe-18.04

Ensuring that my new media server has all the required kernel packages. We are still not done. The IP address of the server has changed, because we now have a different MAC address, so the DHCP server provisioned a different IP. I tried to change the Unifi Controller to provision a static IP address to this new server but I was unsuccessful. I suspect that the new server is also running the Unifi Controller may have something to do with it. Since the IP address has changed, I needed to update the following configurations:

Firewall rules
Unifi Controller name space configurations
Samba configurations because we only allow for local machines to share

All of this took from 4:30pm to 11:00pm last night, 6.5 hours worth of hardware assembly, research with Google, trial and error, and finally success. I cannot imagine if Google and the super helpful community forums did not existed. Fingers crossed that the new media server will run smoothly.

More Home IT Upgrades

This past weekend I continued to upgrade our NAS server. Last weekend, I upgraded my raid array with an additional 8TB of mirrored storage. This yielded two old 4TB WD Blue HDD. I noticed that my case has a total of 9 internal storage bays. One was used by my 500GB SSD Boot Drive, and 6 were populated by HDD drives making up the current raid array. This means I have 2 more storage bays left. However these remaining bays were meant for 5.25″ storage devices like Optical Disc Players. For me to place my old 4TB WD Blue HDD into these bays, I will need a 5.25″ to 3.5″ bay converter. I had one, and purchased the other one on Amazon. I ended up buying the ORICO Aluminum 5.25 inch to 2.5 or 3.5 Inch Internal Hard Disk Drive Mounting Kit.

I also did not have enough SATA slots and purchased the IOCrest SI-PEX40071 SATA III 8 Port Controller Card. This card along with the 4 builtin SATA slots on the motherboard gave me enough SATA connections for my 9 drives.

Once I installed the old 4TB drives, I proceeded to create another md raid level 1 device and created a matching physical volume which I used to extend the current logical volume group. When the setup is completed, I ended up with a 20TB+ fully mirrored NAS server. I love LVM in combination of mdadm.

I figured while everything is fresh on my mind, I minus well proceed with the dreaded 16.04 to 18.04 Linux Ubuntu upgrade.

The upgrade was surprisingly very smooth. However the new version of OpenVPN caused some troubles. The new OpenVPN no longer works with my old PureVPN configuration files, because the certificate files that came from PureVPN used an outdated and deprecated hash algorithm. After getting the new configuration files from PureVPN, everything worked like a charm.

I also have to reinstall the Unifi Controller along with Let’s Encrypt certbot utility.

Super happy with the outcome and the upgrades should last another 2 to 3 years.

Two New 8TB Drive for Our NAS

Our NAS has run out of space again. I saw a deal that the Seagate IronWolf 8TB NAS Hard Drive was on sale at newegg for $309 CAD. I jumped at the chance and purchased two.

I am now following the same step as I outlined in this post. Replacing two old 4TB drives with these two new 8TB drives.

So far so good. Hopefully when all is said and done, my NAS will have a total of 18TB in a RAID 1 configuration of 6 hard drives in total. Two 4TB, two 6TB, and the two new 8TB.

I noticed that I could fit two more drives in my chassis and may decide to re-add the two old 4 TB back in, but first I’ll have to check if my power supply can handle the demand.

I really like this mdadm and LVM setup.

Update: After 2 mdadm syncs, each of which was around 8 hours, and a pvresize that also took another 5 hours. I had to convert the filesystem from 32 bits to 64 bits using these very helpful instructions. Only after I converted to 64 bits can I then expand the existing filesystem to more than 16TB. It was a learning and yet rewarding experience. Next step is to reuse the 2 old 4TB drives in the same chassis and add them to the logical volume.

First Ride with the Wahoo Elemnt Bolt

This week I came a cross a bike computer article and saw this picture. A Wahoo Elemnt Bolt with a gear selection display. Raw emotions took over and I immediately purchased one from MEC.

I longed for a bike computer that can show my current gear selection so that I can avoid looking down and see which gear I am at, or finding out whether I am on the small or the big ring. When I got the SRAM Red eTap two years ago, I totally forgot that it has the ability to communicate its status with a compatible head unit. I was totally okay to continue to use my Garmin Edge 500 until it dies.

The new Bolt unit arrived yesterday. After downloading the companion app on my iPhone it was a breeze to setup and I customized the data pages that I wanted during the ride. It was also super simple to pair the Bolt with my Heart Rate monitor, Power meter, and the SRAM Red eTap.

If you are a data head, then you will love the Bolt. It has more metrics than you know what to do with. I particularly like that fact that it sync’s with my iPhone and when my ride is done it automatically uploads my ride to Strava.

I also allowed the unit to upload health related stats to my iOS Health App. But to my disappointment, it seems that it doesn’t quite sync the calories count. Instead, the Health App got the calories reading from my Apple Watch which has a very skewed reading because I did not turn on the Workout App. I think for my next ride, I will use the Workout App so that the Health App will get a more accurate calories count, and I will turn off the Health App upload on the Bolt to avoid a duplicate recording. I may change my mind later as I am still experimenting which sync’ing technique is best. This is a first world problem, when you have too many health gadgets to manage at once.

As an added bonus, I don’t need to reconfigure my head unit whenever I decide to ride indoors. I can keep my old Garmin Edge 500 to be my indoor bike computer and permanently affix it to my Cervelo R5.

Here is the ride recorded on Strava.

How to Start Road Cycling

What type of cyclist are you?

There are many types of cycling, and each type of cycling comes with their own types of bikes and enjoyment. This web page from CenturyCycles.com does an excellent break down. I love road cycling primarily because of its team work, speed, and relative simplicity in comparison to other forms of cycling. Of course different strokes for different folks. However on this post I am only going to discuss about road cycling since this is the basis of my own personal experience. Here are some fun times which I enjoyed with our local chain gangs.

Getting a bike

Before getting a bike, you will need to know about your own personal dimensions, such as your height, inseam, foot size, etc. These measurements are critical for determining the size of the bike that will work best for you. Different brands can have different sizing metrics. Here is a more in depth article at LiveStrong.com discussing fitting to a Cannondale bike. Below is a table selecting a Trek road bike frame size by your height.

A proper fit is important as you can potentially be riding your bike for hours at a time. An improper fit will result in discomfort and potentially repetitive, body injuries.

Once you know the type and size of the bike that you want to get, you will now have to decide whether to get something new or used. Stay away from buying bikes at major merchandising stores like Costco, Walmart, and Canadian Tires. Those bikes have inferior components and they will not work well and will break within a few rides.

Budget may be an issue here. Be prepare to spend $1500 to $3000 for either a high end Aluminum frame bike or a low end Carbon bike. As a sample, here is a sample link to a local bike store at D’Ornellas listing road bikes that are less than $2000 in price.

Alternatively, you can tap into the local used market. Here is an example from Facebook, a $1200 2015 Specialized Allez Comp. Kijiji is another popular site for locally used road bikes. I personally stay away from eBay, but you may find that more convenient.

I think both Aluminum or Carbon frames have their merits, and I would stay with at least a Shimano 105 group set or higher. The group set is the collection of components on the bike that relates to its transmission system, which is everything from brakes, chain rings, gear shifting, head sets, etc. Here is a quick guide to the different hierarchy of the Shimano road group set from Road.cc.

Pedals and shoes

For road cycling all of us use clipless pedals. There are two main types, the road racing (on left) and mountain biking style pedals. I personally use a pair of Shimano Ultegra Pedals. Pedal selection can be an exercise in finding religion. Instead of me going into details here, I will refer to this wonderful article from the New York City Bike Shop.

The only thing I would stress is that going clipless is a must if you want to keep up with a group. Your choice of the clipless system is really up to you.

Get kitted up

Cycling requires proper attire, primarily for safety, comfort and temperature control. Of course different weather will require different kits (the type of clothing you wear). Things to consider are:

Helmets
Cycling Gloves
Cycling Jersey
Cycling Bibs
Socks

If you decide to ride in the cold, then different linings, jackets, and long tights will also have to be considered.

When considering helmets, you want one that is locally certified for safety. Everyone rides with a helmet now and you will get yelled at on the road if you do not ride with a helmet. Other considerations when selecting helmet is fit, ventilation, visibility, weight, and the style you like. When it comes to bike attire MEC is a good reference for goods and information. Here is short article on helmets.

Personally I am definitely on the practical (or cheap) side when it comes to attire. I prefer fit and function over style and looks. If something works and is cheap then I am all in. I do not really care about brands and colours. I am not keen on any bling bling effects. Having said that, if you do ride in the dark it may be good to invest in some glow and bright clothing, so that you are more visible on the road. I like the sun and try to do all of my riding when it is light and dry.

Accessories

These are the things to have on your bike or on you when you go out for a ride. They fall into two general categories, items that keep your body fuelled and comfortable, and items that look after your bike in case of mechanical issues or road side mishaps and emergencies.

For myself, I usually bring some snacks along for the ride just in case I get into a hunger spell. For rides longer than 1.5 hours, I start bringing high carb content food such as a peanut butter and jam sandwich (cut into bite size units) or a banana. I also love these Endurance TAP maple syrup gels. I usually have one or two in my back pockets.

Aside from energy food, you will also need lots of water to remain hydrated. This means water bottles and bottle cages that fit on your bike to carry them. Everyone is different. You will have to experience how much water you will need. I typically carry a single bottle for rides between 1 to 1.5 hours, and 2 bottles for longer rides. For rides longer than 2.5 hours, I will need to seek an on route water source (eg. a convenience store) to refill the bottles. The amount of water you bring is also weather dependent. You will find that more water is required on more hot and humid days.

For the bike, a multitool to tighten, loosen, or fix odds and ends; one or preferably two spare inner tubes – when (and not if) you get a flat. For quick fills in seconds, a CO2 canister system is the way to go. However, you get infinite air if you use a hand mini pump. Make sure you buy one that can sustain high pressure (90+ psi). A patch kit is great when you run out of spare tubes. I have been on rides when I got more than 3 flats!

Last but not least is a fully charged cell phone so that you can call for help as a last resort or communicate with your riding buddies in case you get separated.

There are other items that people buy to measure their performance and accomplishments. Things like a cycling computer with GPS, a speedometer, and a power meter. I will not go into details here. I personally own a Garmin Edge 500 and it has served me well. All this hard work must be recognized somehow! It also helps to pace a ride.

Your local bike store, or Mountain Equipment Co-op is a good start to procure these accessories.

Join a group

Road cycling is exhilarating, exciting, and will certainly make you more healthy. It gives you more energy when you are not riding, and increase your sense of accomplishment. However, cycling on the road can also be dangerous. In the end, you cannot control who is sharing the road with you and their level of maturity and competence. It is always safer to ride with a group so that you have a bigger presence on the road and you look after each other. So once you are all geared up, join a local club or find some riding buddies in your neighbourhood.

Cycling is always changing. Keep up with the Global Cycling Network YouTube channel.

Improve your riding skills by being attentive, tune your riding skill by learning from each other. Before you know it your group quickly turns into a coherent team. You will pleasantly discover a priceless sense of fulfillment and enjoyment once you are a part of an integral group. You will get there before you know it!

UniFi USG Advance Configuration

In our home we have UniFi networking equipment as our backbone for providing WiFi services. We have also been running a home media and ebooks server. The home media server is running a Plex server, and the ebooks server is running Calibre.

In the past, to access these servers we have to use the following format with the browser:

http://hostname:port

This is not very user friendly especially when you forget the port number. It would really be nice if we can access the servers with something like:

http://media.home
http://books.home

At the same time, the kids have been abusing their screen time privileges. Although their iOS devices can be easily managed with the new Screen Time feature, I still need to govern usage time windows on their gaming PC’s and laptops. I would like to configure our firewall so that Internet traffic from their devices are blocked from 12am to 7am.

The above configurations are both possible on the UniFi Secure Gateway (USG) running EdgeOS. However, these configurations are deemed as advance configurations and are not supported on the web user interface. To perform these configurations, I had to create and edit config.gateway.json file. This file will contain all the key / value overrides of the USG device in a JSON structure. Below are my additions to accomplish the above:

 {
     "firewall":
     {
         "name":
         {
             "LAN_IN":
             {
                 "default-action": "accept",
                 "description": "packets from intranet",
                 "rule":
                 {
                     "3001":
                     {
                         "action": "drop",
                         "description": "Jason Laptop AM",
                         "protocol": "all",
                         "source":
                         {
                             "mac-address": "84:38:35:50:64:12"
                         },
                         "time":
                         {
                             "starttime": "00:00:00",
                             "stoptime": "07:00:00"
                         }
                     },
                     "3003":
                     {
                         "action": "drop",
                         "description": "Kalen Laptop AM",
                         "protocol": "all",
                         "source":
                         {
                             "mac-address": "e0:ac:cb:8c:32:00"
                         },
                         "time":
                         {
                             "starttime": "00:00:00",
                             "stoptime": "07:00:00"
                         }
                     },
                     "3005":
                     {
                         "action": "drop",
                         "description": "Jason Gaming AM",
                         "protocol": "all",
                         "source":
                         {
                             "mac-address": "1c:1b:0d:70:80:84"
                         },
                         "time":
                         {
                             "starttime": "00:00:00",
                             "stoptime": "07:00:00"
                         }
                     },
                     "3007":
                     {
                         "action": "drop",
                         "description": "Kalen Gaming AM",
                         "protocol": "all",
                         "source":
                         {
                             "mac-address": "1c:1b:0d:72:ed:bf"
                         },
                         "time":
                         {
                             "starttime": "00:00:00",
                             "stoptime": "07:00:00"
                         }
                     }
                 }
             }
         }
     },
     "system": {
         "static-host-mapping": {
             "host-name": {
                 "media.home": {
                     "alias": [
                         "media"
                     ],
                     "inet": [
                         "192.168.168.9"
                     ]
                 },
                 "books.home": {
                     "alias": [
                         "books"
                     ],
                     "inet": [
                         "192.168.168.9"
                     ]
                 }
             }
        }
    }
 }

The above config.gateway.json file had be stored in the directory /var/lib/unifi/sites/default on my Ubuntu box that is running the UniFi Controller software.

The next step is to perform a force provision of the configurations to the actual USG. This can be done with web based controller software.

To double check, I can run the following on the actual USG box itself.

mca-ctrl -t dump-cfg > config.txt
less config.txt

Any JSON path key and its corresponding value can be overridden using this technique.

The USG is truly a hidden source networking gems.

HomePod Setup WiFi Woes

I’ve setup our Apple HomePod plenty of times. All previous setups have gone without any issues.

Tonight I had to reset our HomePod and set it up again because Siri was unresponsive. However, this time around the setup told me that I had an Incompatible WiFi, WTF!

After much Internet research I finally found the problem. I setup my WiFi via a Profile that I created and signed. This apparently was the issue. After I forget my network and rejoined by physically typing in my password, the setup went without any problems.

This may be an edge case. But for those who fell into the same rabbit hole, I hope this blog entry serves you well!

Hello Koodo

Our family was on the Rogers Share Everything Plan. We paid $85 for the primary line and $35 for four additional lines. This is a total of $225 for 6GB of shared data per month.

As our boys use their phones more and more both during the weekend and at school, the 6GB data cap is being surpassed more and more frequently. I wanted to shop for a plan that has more data, preferably staying with Rogers. Rogers has a promotion for 14GB (6GB original) plan for $105, but each supplemental line costs $45 each resulting in a total of $285.

This is an effective increase of $60 per month. Surely with modern advances in data throughput and capacity other carriers can do better. I decided to shop elsewhere.

I first looked at Chatr and Public Mobile. Both give you nice bargain data rates but are limited to 3G speed, something I did not want to entertain.

From Public Mobile’s web site I discovered the possibility of Koodo and found that their $50 per month plan was perfect for us. This will increase our data capacity to 4GB per line. This is 20GB in total although not shareable, which I think we can live with. Another pleasant surprise is that when we activated our new plan in the store, we got a $10 per month discount for each line for the first 12 months! So instead of $250 per month, we are now looking at just $200. For $25 less we enjoy more data per individual!

I am quite happy with this transition. I’ve included a QR code of my referral code as a banner of this post, so that if you decide Koodo is right for you, then use my referral code. This way we can mutually benefit. Make sure you use the referral code before activating your new line. Unfortunately they spelt my name wrong, but the code should still work. Enjoy!