Rogers All Day Outage

Today we experienced the longest residential Internet outage that I have personally experienced since I have been a Rogers customer for more than 20 years.

My firewall records show:

As per the records above, one can observe that initial instability started at around 2:15am and then at around 4:51am it failed for a very long time. There were attempts at around noon and in the evening to resurrect the system. These attempts proved to be unsuccessful. In each of these attempts, there were issues with domain name lookups, and possible routing problems.

Our Koodo LTE on our mobile devices were able to make phone calls, but data rates were very slow. Usability was intermittent at best. To get the latest news, we had to resort back to our emergency radio.

We found out that this radio’s dynamo generator no longer works, and we should probably replace it soon. We tried solar but finally settled for batteries to operate the radio.

It turns out the whole country is impacted with specific incidences of:

• Point of Sales (POS) terminal issues with using and clearing debit cards;
• Work at home businesses can no longer function;
• Issues with ArriveCan applications for those entering the country;
• Possible 911 reception issues in certain areas;
• This Toronto Star article has more.

I am writing this post at around 10:15pm. At this point the Internet is backup now longer than any of the previous Rogers attempts in fixing the outage. I hope we are up for good.

My speculation is that Rogers’ provisioning database was hacked and got corrupted causing the disablement of Internet access by all of their customers. This is my GUESS. I have no proof. We will see what really happened if Rogers choose to divulge such information in the near future.

There has been precious little information at this point on what caused the outage. My curiosity is piqued to find out what happened, and how Rogers can prevent this in the future.