DuckDB – in-process SQL OLAP database management system

Re:DuckDB – in-process SQL OLAP database management system by jakewins on Friday February 10, 2023 @12:27PM (#34742750)

I used it as glue for a monthly job, last week, processing ~1B event records, juggling them a bit and pushing them as 2M parquet segment files to S3.

Mixed experience, would definitely not put it in a system that isn’t ok crashing frequently.

It segfaults on Alpine (argh, C++!), and force exits the whole NodeJS process when it gets unexpected HTTP responses from S3.

In an archive run of 2M pushes it’ll crash the process 4-5 times.

Overall still really, really like it, but learned to not trust it

[ Parent | on HN ]

Re:DuckDB – in-process SQL OLAP database management system by tracker1 on Friday February 10, 2023 @03:52PM (#34745670)

Node + AWS for data ingress has been pretty painful in my experience (mostly dynamo feeds from large csv (whois database)). In the end, rewrote in C# (core 2) and it was able to complete more reliably. I'm guessing that go and rust would also be better. I like node, really like JS, but I just think that maybe the AWS libraries aren't that great in the space. It would run for 3-5 hours, then just blow up unexpectedly, even with plenty of memory overhead, and not really bandwidth limited, with appropriate retries and slowdown for dynamo rejections.

If I never have to write ETL pipelines again, I won't be upset about it.

[ Parent | on HN ]

Re:DuckDB – in-process SQL OLAP database management system by Master_Odin on Friday February 10, 2023 @04:52PM (#34746428)

There was a bug that was recently fixed in node 16.17+ that was causing hard node processes crashes when doing stuff with S3 and I think had to do with receiving multiple packets at once or something.

[ Parent | on HN ]