Unifi Access Point Disconnected

I have the following Unifi networking setup:

  • Unifi Secure Gateway (USG-3P) running firmware
  • 3 Unifi Access Points AP-AC-PRO running firmware
  • 3 Unifi AP-AC-M running firmware
  • Unifi Controller (6.2.25) running on Ubuntu
  • The AP’s are connected to Unifi Switches running firmware

The AP’s are separated into two AP Groups. One of the AP Groups contains the 3 AP-AC-PRO and one of the AP-AC-M. This latter group was the problematic group. The other AP Group which contains the remain 2 AP-AC-M continue to work flawlessly throughout incident. From here on when I reference an AP Group, it is the problematic group that contains the 3 AP-AC-PRO and 1 AP-AC-M.

It all started yesterday when I noticed that my WiFi was a bit slow in the backyard and I wanted to change my radio configurations on one of the AP-AC-PRO and one AP-AC-M in the same AP Group. After the provision, the AP went disconnected as shown by the controller.

I ssh into the problematic AP-AC-PRO and discovered in /var/log/messages that there were many instances of the following log entry:

syswrapper: [state is locked] waiting for lock

I attempted to reboot the device but the device remained in the same “locked” state.

Since it is inconvenient for me to physically reset the AP, I attempted to reset the device via ssh using the command:

syswrapper.sh restore-default

Unfortunately, this did not always work because it immediately just shows the same locking message:

syswrapper: [state is locked] waiting for lock

I had to reboot the device via the command line reboot. As soon as I can ssh into the device after reboot, I immediately execute the restore command as above. This took a lot of trial and error because my timing is often off. When I miss the window, I will get the lock message again. I find that my chances are higher if I first forget the device on the controller first.

A quick suggestion to the Unifi team. It would be nice that the restore-default command if not able to restore immediately due to the lock, would at least set a flag in persistent flash memory of the device so that on the next reboot it will perform the restore then. This feature will safe me A LOT OF TIME!

Once the device is reset to factory default I proceeded to reconfigure it to the WiFi networks that I had. Unfortunately, when I try to provision the changes (adding the re-adopted device back into its original AP Group that is associated with the WiFi networks), it went into the disconnected state again. To make matters worse, the other AP’s in the same AP Group started to misbehave. Some would go into a provisioning state and followed by a disconnected state, while others go into an adopting state. This is of course very unnerving and frustrating. However, this observation lead me to remember a previous episode that I experienced a few weeks ago.

When I updated the controller to 6.2 and upgraded the AP’s firmware, the AP exhibited a similar locking issue. The solution that I employed was to restore to factory default and re-adopt the device. However after readopting, I assigned the AP to a brand new AP Group which I associated with the original WiFi networks. Simply adding the device to the original AP Group did not solve the issue.

When I tried this solution yesterday, it did not work. The device continues to go into a disconnected state immediately after provisioning when I added to the new AP Group. After many hours and much experimentation, I decided to erase all the WiFi networks and the problematic AP Group. I recreated the WiFi networks, and created a new AP Group and proceeded to add each AP one by one (after a reset to factory default). In summary here are the final steps that got me out of this pickle:

  1. Forget all AP’s in the affected AP Group.
  2. Remove all WiFi networks from the AP Group.
  3. Delete the AP Group.
  4. Delete all the WiFi networks that was associated with the above AP Group.
  5. Re-create all the WiFi networks and associate with a brand new AP Group.
  6. For each AP, use ssh to reset to factory default, adopt, and add them one at a time to the AP Group using the controller web UI.
  7. Since there were four AP’s (3 AC-Pro and 1 AC-M), I waited until the AP is fully connected and can service WiFi clients before I continue with the next one.

I am documenting this so that I can share with Unifi support. This has happened twice now, and each time I spent multiple hours to try to get my WiFi network working. In these pandemic times, WiFi is as important as electricity and plumbing. Since Tier-1 support was unable to resolve this issue, waiting for Tier-2 support (around 24 hours) is a bit “hard to swallow”.

Any ways I am glad that I was able to resolve this and brought my WiFi networks back up and running with the 4 affected AP’s, FOR NOW. However I must admit, these two episodes have made me apprehensive of making configurations to these AP’s, thinking that the next provisions will result in many more lost hours.

I hope the Unifi team can use this information and see if there is an issue relating to AP Group provisioning, since this seem to have triggered the issue in both cases.

Green Sunroom Project

YouTube viewing has been one of our favourite pass times during the lock down nature of the Covid-19 pandemic. I personally have been watching quite a few channels on how to use LiPO4 cells to build rechargeable battery banks for solar applications, primarily for off grid purposes.

We have a sunroom in our back yard that we used during the summer to grow some vegetables. It has some electrical needs such as water pumps, a temperature sensor, and a fan. Currently there is an electrical socket, fed from the house, that we plug these devices into. We thought it would be a good project to try to get our sunroom off grid. This would be a good learning project.

The first task is to build a 12V LiFePO4 prismatic cells battery bank. I purchased 4 3.2V 100Ah battery cells from AliExpress. The cells came with bus bars so I did not have to purchase those. However, I did have to buy a battery management system (BMS) to balance and manage the charging and discharging of the battery cells. It was very tempting to buy a BMS from AliExpress, but I decided to be cautious and purchased one from a US vendor with the accompanying and preferred quality control. The company Overkill provides a 12V BMS specifically for four LiFePO4 battery cells in series.

It took a very long time for the batteries to arrive from China. I suppose the pandemic could be one of the many reasons for the delay. Once they arrived, I connected in parallel and proceeded to perform a top balance procedure with my voltage limiting desktop power supply. This step is required because each cell will have a different voltage potential from each other. We want all the cells to have the same voltage potential to maximize the capacity that we will get from the aggregated 12V battery bank.

Cells in parallel being topped balanced at 3.65V until zero current

To top balance all the cells, first I hook up the cells in parallel and charge them at a constant voltage of 3.65V. The charge will continue until my desktop power supply shows zero amp going into the battery. This process took a very long time, almost 2 days.

Once the cells are balanced, I reconfigured the cells in series and proceeded to hookup the BMS and the pure sine wave 600W inverter I purchased from Amazon. I had to buy 4 AWG wire, once again from Amazon, because the 10 AWG wire that I purchased earlier was not going to be enough if I want to discharge the battery at 600W which is going to result in more than 50A of current at 12V. I used the remaining 10 AWG wire for solar controller and panel hookups. I also got some XT90 connectors so that I can easily plug/unplug the solar charge controller, solar panels, and potentially plugin charger. I will talk about the solar side some more later on.

All wired up. The yellow XT90 connector is to either a solar charge controller or an external DC charger

So now that we have the guts of our 12V LiFePO4 battery pack, we need to find a suitable home for this thing. My wife had an extra plastic filing box hanging around which is perfect for this.

A filing box is perfect to fit everything
Custom grommets and added a PC fan

I needed to drill some holes to fit a 12V 120mm PC fan for ventilation, and a couple of 2″ grommets so that we can pass plugs and connectors through the box. The fan will be powered by the inverter.

At this point we have ourselves a 1200Wh portable super battery pack that can power up to 600W of electronics, which will be great for road trips. If you plug a 20W iPhone fast charger and charge your phone, it can continuously charge for 60 hours (2.5 days). That is a lot of phones. If your MacBook Air ran out of juice on the road, then this battery pack can power a 45W charger for your MacBook Air for more than a day, and also charge your computer fully. Quite a handy thing to have for emergencies.

Doubles as a 1200Wh portable battery bank

For the solar panels, I purchased two Xinpuguang 100 W flexible solar panels from Aliexpress. They were about $1 / Watt, a pretty good deal. I hook the two panels together in series and got a Victron BlueSolar MPPT 75/10 solar charger to manage the charging of the batteries. The charge controller can accept a maximum of 75V and outputs a maximum of 10A.

The charge controller will automatically adjust the amperage and voltage to the battery bank as required ensuring optimal charging scenario. During a sunny day, it will run the sunroom load from the panels and any remaining current will goto charge the battery. At night, the battery will run the sunroom.

Today, we installed the entire setup. The battery is placed inside the green house to give it some precipitation protection.

The panels are latched to the roof of the green house, one on each side.

The BMS unit has a bluetooth connection and an iOS App. I can use my iPhone when in bluetooth range of the battery to see if the battery is being charged or discharged.

I took the following screen shot of the app today at around 5pm EDT. You can see that there is no current going into the battery and no current going out of the battery. This means the sun is powerful enough to run all the pumps and other electrical appliances in the sunroom. Pretty cool!

It is still too early to tell yet whether there is enough sun power to charge the battery and run the electrical devices in the sunroom in a sustainable manner. My current suspicion is that the two panels are just enough even on a full, bright, sunny day and at peak hours, to power devices and also provide surplus current to charge the batteries.

Here is my overall connectivity diagram:

We will let the system run for about a week to see if this is sustainable during the summer months or not. If not, then I will have to create an automatic transfer switch so that we can intermittently recharge the batteries during the evening with an optional 480W DC charger, which I also got from Aliexpress. This charger can operate between 0-24V and 0-20A. To charge the battery bank, I have set it to a constant voltage of 14.0V and allow the output current to flow unrestricted. This should charge the battery fully in a little over 4 hours from scratch.

Overall, I learned a lot from this project and what a great way to spend the pandemic indoors. This could be a precursor to a DIY Tesla Powerwall Project. We’ll see.

I See the Light!

Today my wife and I got our first dose of Pfizer BioNTech Covid 19 Vaccine after probably more than 14 months since the first lock down notice from 2020. Of course it is still too soon to declare victory, as our second dose is still scheduled 3 months from now.

On a related note, the Ontario government has also declared the AstraZeneca vaccine will no longer be available as a first dose: “the decision was made out of an abundance of caution.”

CBC Article on AstraZeneca

I can’t help but think that this sounds a bit contradictory to what the government was peddling a few weeks ago. It kind of makes you think whether their 3 months guidance between the two doses of mRNA based vaccines is warranted and backed up by facts or not.

Regardless, this is another case of inconsistent information provided by our institutions. It certainly will continue to chip away at the credibility of the same institutions.

I hope we learn something from episodes such as this. In the meantime, both of our sons, who are age 17 and 16 should also be eligible for Pfizer or Moderna soon. We are now just monitoring our local clinic schedules to see when they can be booked. Fingers crossed, and three months cannot come soon enough.

The story continues in the media:

More updates from the CBC

Recent Gout Attack

On the Wednesday morning of April the 14th, right after I took out the garbage, I noticed that I experienced some tightness when I was bending my left knee going up the stairs. As time goes by the knee begins to swell. By lunch time, it was swollen to the point that I can no longer bend my knee and I can feel a continuous throbbing pain around my left knee. I can also feel that the temperature around my knee has also been elevated. When I took some measurements with a touchless temperature sensor for measuring fevers, my left knee is like 2ÂșC more than my right knee. This is not my first gout attack, but it has been very long since my last one that I couldn’t even remember when I had it.

I decided to document my current experience, so that next time, I personally have something to recount. Therefore, this blog entry is more for myself than you the reader if you are reading this already.

I have had swollen joints due to gout in my ankle and my knee, always on my left side for some reasons. I wasn’t sure whether I should take Allopurinol or wait until the attack subsides. I seem to recall that in the past that I should not take any Allopurinol during an attack, but the recollection was pretty murky, and doctors are hard to come by these days, so I read the following article: Allopurinol Doesn’t Worsen Acute Gout Attacks, and decided to take Allopurinol in my medicine cabinet. Later on I found out from my family doctor and my local pharmacist that I may have been better off not taking it. What I can tell you is that it started to heal even with the taking of Allopurinol during the attack, so there is some truth to the above article.

I also decided to take Ibuprofen which is available over the counter to reduce the inflammation. The Ibuprofen worked within a couple of hours and really helped with the throbbing. In the end, there is not much to do but to wait and drink lots of water and let my system dissolve and eat away at the uric crystals in my knee.

I got some diclofenac sodium topical cream, which I applied on the affected area overnight. Unfortunately, I felt some skin irritation, and an over tightening or stretching of the skin situation. I had to wash it out in the morning and stuck to Ibuprofen. A single does of 400mg Ibuprofen didn’t do the trick. I had to double the dosage to 800mg.

My hope of the swelling going away in a few days was dashed on the first weekend when I knew that it was getting worse and not better. It was not until the second weekend (approx. 10 days into the ordeal) when I can finally begin to bend my knee. At this point I stopped taking the Ibuprofen and let it naturally take its course.

By the third week, I can bend my knee for most daily activities but going down the stairs and simple things like putting on my pants still created a sharp pain sensation.

This has been the longest gout attack that I have experienced. This time around, I noticed significant muscle atrophy in my left thigh and calf. I also noticed pull tendon or cramping sensations when going down the stairs. Fortunately these sensations went away as I started to do more walking in the latter part of the third week.

On day 22, I still have minor pain on my knee cap but I was able to walk so I did a 3.5 km walk with my wife in the evening of May 6th. Since there were no additional swelling the following day, I decided to do an indoor bike ride of 20km. Yesterday was another 3.5 km of walking and today another indoor ride.

As you can see from the Strava statistics, I am improving, so back in the saddle and back on the climb. Slowly but surely.