Re:How we scrape 300k prices per day from Google Flights
by mongodbhater on Saturday June 13, 2020 @02:55PM
(#23511720)
All the (AWS) technologies used are totally unnecessary. SQS/DynamoDB/Lambda. I can buy a laptop in walmart for $500 and i can do all the scrapping in starbucks wifi.
[ Parent
| on HN ]
Re:How we scrape 300k prices per day from Google Flights
by rblatz on Saturday June 13, 2020 @03:02PM
(#23511779)
Right, it seems like they overbuilt this hacky solution. You are scraping, eventually you just need to subscribe to the data. Why invest that much effort into a temporary solution.
Re:How we scrape 300k prices per day from Google Flights
by nunez on Saturday June 13, 2020 @08:06PM
(#23513959)
Lambda is needed to get rotating IPs and scale while avoiding browser fingerprinting. SQS takes the results of those scrapes and puts them into a database, DynamoDB. It's a straightforward web scraping pipeline.
Re:How we scrape 300k prices per day from Google Flights
by toddh on Tuesday June 16, 2020 @12:20PM
(#23540955)
Lambda isn't enough. You'll get blocked in a heartbeat. You still need a proxy service.