News for hackers. Stuff that matters.

A comment on DuckDB over Pandas/Polars. Parent comment

Re:DuckDB over Pandas/Polars by DiscreteTom on Tuesday November 05, 2024 @09:29PM (#42056865)
I tried to spread large dataset into thousands of files on S3 and use StepFunctions Distributed Map to launch thousands of Lambda instances to process those files in parallel, using DuckDB (or other libs) in Lambda. The parallel loading and processing is way faster than doing this in a single big EC2 instance.
[ Parent | on HN ]
Re:DuckDB over Pandas/Polars by ramraj07 on Tuesday November 05, 2024 @09:57PM (#42056963)
Lambda isn’t infinitely parallel. I thought it doesn’t do more than 100 parallel runners? I4i.metal has 96 cores and can be faster than that.
[ Parent | on HN ]
Re:DuckDB over Pandas/Polars by DiscreteTom on Wednesday November 06, 2024 @09:56PM (#42072882)
As per AWS said in https://aws.amazon.com/cn/blogs/aws/aws-lambda-functions-now...

> Each synchronously invoked Lambda function now scales by 1,000 concurrent executions every 10 seconds.

[ Parent | on HN ]