News for hackers. Stuff that matters.

A comment on Show HN: I added public H-1B data to my automated job board.

Re:Show HN: I added public H-1B data to my automated job board by dgs_sgd on Tuesday July 16, 2024 @02:11PM (#40978850)
How long does it take to run your scraping pipeline?
[ Parent | on HN ]
Re:Show HN: I added public H-1B data to my automated job board by james_dev_123 on Tuesday July 16, 2024 @04:33PM (#40980071)
End-to-end it takes ~10 hours. The biggest bottleneck is the actual scraping itself. The GPT part is relatively quick.
[ Parent | on HN ]