Automating tasks using a web crawler

I’ve recently developed a deep fascination with web crawlers/scrapers. Despite knowing about them for a while, I’ve never had any real need or desire to develop one until recently. This change was brought about by the gruelling job application process that I have been engaging in recently.

As I made the decision to sign up to multiple online job search platforms, I’ve become familiar with some of the shortcomings of each site. However, instead of sending the devs an email and hoping for the best, I decided that I would take matters into my own hands wherever I can and use the tools I’m equipped with in order to solve these issues.

However, there is one tiny pet peeve about this platform — I don’t get notified when my application status changes. This means I have to constantly load the job applications page in order to get an update. This isn’t optimal for me as I have to send out applications on other sites and build a decent portfolio. The solution? Build a web crawler that will generate the notifications for me! It was a fairly simple solution that saved me tonnes of time.

Because I have a penchant for sub-par puns, I decided to name it “wobbot”. Anyway, I digress. The rest of this article will go into the technical details of the bot.

The code in this article is written within a “main” function and is called as shown by the code below:

Once we’re logged in, we navigate to the job cards section and open the “I’ve applied” tab on the page. This is where all the applications that have been sent out live. Once here, I stumbled upon a minor challenge. WOBB only displays 5 jobs at a time, at the time of writing, I had 27 jobs in this section. Selenium, however, does not allow for interaction with hidden elements. The only way I could interact with all the cards would be to make all of them visible first.

Now comes one of the more interesting parts — for each card I retrieve, I will extract the job title, the posting company, the application status and the date applied. There was no apparent way to get the unique job id so database queries had to used the “and” operator with all the details other than status to uniquely identify different jobs. I did this with the faith that no 2 postings will have the same title, company and date. A bit of a risk, but I was willing to make that assumption.

Related posts:

This documentation aims to answer the following questions as I believe that sufficient understanding of the answers will bring ease in creating a useful and usable Quiz feature. The target users are…

From the time I began learning Vietnamese at home and English in public school, I have lived with dual identities pulling me from seemingly opposite ends of a spectrum: Refugee and American. Lower…

The struggles of being a young person in the world of health and safety can be numerous and varied. For one, young people may not have the same level of experience and knowledge as older individuals…