News for hackers. Stuff that matters.
Re:How we scrape 300k prices per day from Google Flights by mongodbhater on Saturday June 13, 2020 @02:55PM (#23511720)
All the (AWS) technologies used are totally unnecessary. SQS/DynamoDB/Lambda. I can buy a laptop in walmart for $500 and i can do all the scrapping in starbucks wifi.
Re:How we scrape 300k prices per day from Google Flights by rblatz on Saturday June 13, 2020 @03:02PM (#23511779)
Right, it seems like they overbuilt this hacky solution. You are scraping, eventually you just need to subscribe to the data. Why invest that much effort into a temporary solution.
Re:How we scrape 300k prices per day from Google Flights by nunez on Saturday June 13, 2020 @08:06PM (#23513959)
Lambda is needed to get rotating IPs and scale while avoiding browser fingerprinting. SQS takes the results of those scrapes and puts them into a database, DynamoDB. It's a straightforward web scraping pipeline.
Re:How we scrape 300k prices per day from Google Flights by toddh on Tuesday June 16, 2020 @12:20PM (#23540955)
Lambda isn't enough. You'll get blocked in a heartbeat. You still need a proxy service.
Re:How we scrape 300k prices per day from Google Flights by atesti on Sunday June 14, 2020 @03:44PM (#23520702)
No, the access google over residental proxy servers provided by packetstream.io
Re:How we scrape 300k prices per day from Google Flights by sdinsn on Saturday June 13, 2020 @09:20PM (#23514336)
Of course its unnecessary. The point is that you can do it in the cloud, instead of on a laptop at starbucks...