News for hackers. Stuff that matters.
Re:How we scrape 300k prices per day from Google Flights by nunez on Saturday June 13, 2020 @08:06PM (#23513959)
Lambda is needed to get rotating IPs and scale while avoiding browser fingerprinting. SQS takes the results of those scrapes and puts them into a database, DynamoDB. It's a straightforward web scraping pipeline.
Re:How we scrape 300k prices per day from Google Flights by toddh on Tuesday June 16, 2020 @12:20PM (#23540955)
Lambda isn't enough. You'll get blocked in a heartbeat. You still need a proxy service.
Re:How we scrape 300k prices per day from Google Flights by atesti on Sunday June 14, 2020 @03:44PM (#23520702)
No, the access google over residental proxy servers provided by packetstream.io