Ask HN: Those making $500/month on side projects in 2024

Re:Ask HN: Those making $500/month on side projects in 2024 ... by Jabbs on Tuesday December 10, 2024 @11:50AM (#42378571)

Selenium, although I'm using a wrapper library that uses it. I only query each company every few days or so which probably helps to not get banned IP-wise but also rotate them. But many of the company job links are through external sources too (lever, greenhouse, etc.) which don't seem to mind

The company data was gathered online for a long time until I found https://www.thecompaniesapi.com/ (which now is the source for much of that data)

[ Parent | on HN ]

Re:Ask HN: Those making $500/month on side projects in 2024 ... by stuckkeys on Tuesday December 10, 2024 @01:31PM (#42379721)

I tried to use my own desktop machine to process some of these tasks. I can see my fans go jet mode when the scraping was being done lol. Do you have discord or any way to connect? Would love to chat around this topic. Feel free to drop any social media handles. I’ll ping you.

[ Parent | on HN ]

Re:Ask HN: Those making $500/month on side projects in 2024 ... by Jabbs on Tuesday December 10, 2024 @02:00PM (#42380130)

Ohhh yea I run into this memory issue very quickly when scraping (especially if you have a large URL dataset then it will inevitably find a website with a giant bit of markup). So I have to set timeouts and blacklist timely requests but also completely reset the (headless) browser on 2-3 requests (which is overkill but I am restricted on memory for those workers). Feel free to drop me an email sometime (should be on my HN profile)

[ Parent | on HN ]