You must have heard about the term “Web scraping” a lot if you have been in the software development Industry. I knew what it is but never actually understand its use case and what is it capable of. I read about puppeteer a long time ago and got excited to see what it can do like but never had a real use case for it. I found a use case, but there is a story for it, stay with me on this.

Recently my brother who works in a travel agency is talking about appointment dates for passengers. So the gist is all the travel agents try to get the latest appointment dates for passengers traveling to countries and whoever gets the earliest dates get paid more. My brother spoke about the pain of how he keeps on checking his phone every 15 mins like logging in and filling in all the details and getting the latest date.

The earliest dates get opened at any time of the day or night. He was telling me about all this, and then I got an idea what if I set up a bot that logs in fills in all the details, and gets you the latest date for the particular country every 10 mins? And what if every 10 mins I send him a WhatsApp message of the latest date and if he sees the earliest date he can just go to the site and book it?

That was it, I did the same, I know setting up a bot sounds like a very complicated thing but it wasn’t. Firstly I wrote what are the things that I had to do like

• Set up puppeteer

• Scrape the website to login

• Click on the dashboards on the booking page

• Fill in the appointment details

• Get the earliest dates

• Setup WhatsApp-web.js

• Send message

After the puppeteer setup, scraping to log in was a bit challenging as I was already getting to know puppeteer, I must say there is a good learning curve to it. However, I managed to log in but there was a catch if you log in for like more than 6-7 times it blocks your IP for 2 hours. I was stuck on that for like more than 2-3 days thinking about whether I have to set up a proxy or VPN or some way to change my IP every time I log in.

I almost felt like giving up on it, you know when it comes to your mind what's there to lose if you don't do this? But I thought let's keep scraping and look into this later. So I kept on scraping with limited attempts and waiting for 2 hours after every 7-8 attempts of running scripts. Once I got to the appointment details page and get my earliest appointment date it Felt like a yes! I did it. But my mind instantly felt like, Wait there are a lot of things to solve. How will I send the WhatsApp message? What about this IP-blocking thing? Do I have to Run the scripts overnight all the time?

I kept going and implementing small things and learned how to send WhatsApp messages and save the QRcode authentication to be persisted every time you re-run the script. I looked for different proxy services and other things but that IP issue didn't solve. Then I thought What if I don't log out? And stay logged in and keep a function to call every 10 minutes that gets me the latest date. There was a challenge to it every 15 mins the session expiry popup comes up but I cracked it with puppeteer and stay logged in.

Finally, I set up all the things and my brother is now able to book appointment dates earlier than any Agent in India. Although there is a financial benefit to me in this, Apart from that as a developer I feel when you are doing your job or whatever you get into your comfort zone and stop learning and upgrading yourself. Solving this kind of problem actually feels like you have some power or some kind of edge in life. It took me 2-3 weeks to build this along with my job. Manier times thoughts come like giving up, getting stuck somewhere where you don't get an answer to it.

Hours and hours of debugging head scratching but everything does fall into place once you are persistent on it. This may be one of the things that people say about solving real-world problems, I know this isn't too big or scalable but it feels nice when you can do it. One of the key reasons for writing this post, In the future I will remember how I solved a problem that was intimidating at first and painful throughout but I did it despite all of it.