It looks like York Region is beginning to ramp up our vaccine schedules. We are now awaiting our postal code to be eligible so that my wife and I can make an appointment. At the time of writing this post, unfortunately, our postal code is still not eligible.
We know some of our neighbours are checking the website, https://york.ca/covid19vaccine on a regular basis. Obviously, this is very tedious and frustrating. I decided to embark on a small project to detect our postal code from the web site. My goal is to write a script that can be executed from my NAS server every 30 minutes to see if it can find our postal code on the website. My first attempt was to use Python and Selenium. The API was a bit clunky and I found it to be somewhat unreliable and fraught with timing issues. I gave up on this approach.
I thought I give it another shot, but this time using Puppeteer. This was much more simpler to use, and I found it to be more friendly especially if you’ve worked with Browser based Javascript to begin with.
I wrote this simple node script called scrape.js:
#!/usr/bin/env node
const puppeteer = require('puppeteer');
const targetPostalCode = process.argv[2] || 'A2B';
console.log ('scraping for ' + targetPostalCode + '...\n');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('http://york.ca/covid19vaccine');
const anchorSelector = 'div#Y_accor div.new h3.Y_accordion span.trigger a';
await page.waitForSelector(anchorSelector);
console.log ('received...\n');
const result = await page.evaluate((anchorSelector) => {
let res = [];
document.querySelectorAll(anchorSelector).forEach(function(elem){
if (elem.textContent.match(/.*50 Years of Age and Older.*/i) != null) {
elem.parentElement.parentElement.parentElement.querySelectorAll("p").forEach(function(elem){
if (elem.textContent.match(/.*postal codes.*/i) != null) {
const postalCodes = elem.parentElement.querySelector("strong").textContent;
postalCodes.split(", ").forEach(function(pc) {
res.push(pc.trim());
});
}
});
}
});
return res;
}, anchorSelector);
console.log(result);
let found = false;
let regex = new RegExp(targetPostalCode, "i");
result.forEach(function(postalCode) {
if (postalCode.match(regex) != null) {
// Postal Code Found!
console.log("We found: " + targetPostalCode);
found = true;
}
});
console.log('closing...');
await browser.close();
})();
I placed this script in the directory /home/xxxxx/gitwork/covidMonitor. Note that I targeted “50 years of Age and Older“, because this is the age group that I am personally interested in. The A2B is also not my real postal code. It is used instead of my own private postal code.
Before I run the program, I had to install puppeteer in my working directory.
cd /home/xxxxx/gitwork/covidMonitor sudo npm i puppeteer
Running the above scrape.js produces the following:
scraping for A2B...
received...
[ 'L0J',
'L4B',
'L4E',
'L4H',
'L4J',
'L4K',
'L4L',
'L6A',
'L3S',
'L3T',
'L6B',
'L6C',
'L6E' ]
closing.
Of course A2B is not in the list. If it was, another line like:
We found: A2B
will be in the output.
I supplemented the above with a shell script program which I can later use to configure cron to be executed every 30 minutes. The shell script’s job is to monitor the website using scrape.js and if the postal code is found, then send a notification email.
#!/usr/bin/env zsh
cd /home/xxxxx/gitwork/covidMonitor
if [ $(./scrape.js | tee /home/xxxxx/log/covidPostalCodes.log | grep "found" | wc -l) -ge 1 ]
then
sendmail xxxxx@gmail.com <<@
Subject: Covid Vaccine Can Be Scheduled
Goto the following site to schedule your vaccine:
https://york.ca/covid19vaccine
@
fi
I could have included the email notification logic in the node script as well, but I figure I may want to check manually on an infrequent basis, in which case I simply run the scrape.js script on the command line.
Now all there is to be done is relax and wait, a thing that we have been doing a lot lately during the pandemic.