Re:How we scrape 300k prices per day from Google Flights
by nunez on Saturday June 13, 2020 @08:06PM
(#23513959)
Lambda is needed to get rotating IPs and scale while avoiding browser fingerprinting. SQS takes the results of those scrapes and puts them into a database, DynamoDB. It's a straightforward web scraping pipeline.
[ Parent
| on HN ]