News for hackers. Stuff that matters.

sgignoux

sgignoux on Hacker News

Ask HN: Best practice for in-memory large data transformation

Posted by sgignoux on Tuesday September 27, 2022 @05:55AM
from the ask-hn dept.

Does any of you have some recommendation on how to set up an in-memory data transformation system. I have been looking at DuckDB[1], but maybe there are interesting alternatives. Apache Arrow seems promising in some ways too.It is about running a series of transformations, defined in SQL, on many tables, but:- All the data can fit in the main memory of a single server (a few TB)- No change in...

( Read More... | no comments | on HN )
Re:Xilinx-Samsung SmartSSD Computational Storage Drive Launched on Friday November 13, 2020 @12:22PM (#25084704)
That shouldn't be too hard to get a SQLite-ish system working on the "smart" part of the SSD. What would be the uses cases that you have in mind?
Re:Why just saying “fail fast” is terrible business advice on Thursday November 06, 2014 @12:49PM (#8568439)
Thanks for the feedback. Thinking about it, what I wrote is probably quite obvious to the usual crowd of Hacker News.
Re:You’ve got an idea for a startup, now what? Seven actions to create... on Wednesday April 27, 2011 @02:51PM (#2490608)
I still think that some thinking and planning are important, but I agree definitely that speed of implementation is a critical success factor for startup. The shorter your build-and-ship cycle is, the more feedback you can get. In the worst case, if you build something that is not as useful as you expected or that has a negative impact, you didn't waste a lot of time before correcting your trajectory.