Re:I wrote one of the fastest DataFrame libraries
by cfourpio on Sunday March 14, 2021 @12:36AM
(#26453057)
Would you mind pointing me at a resource that explains how J/Jd handle it by comparison?
Re:I wrote one of the fastest DataFrame libraries
by rscho on Sunday March 14, 2021 @01:52AM
(#26453342)
https://code.jsoftware.com/wiki/Jd/Overview
Jd only packs int vectors, though. So if you're hoping for string compression then I don't know of any free solution. Jd heavily leverages SIMD and mmap. Larger-than RAM columns can be easily processed by ftable. I use Jd for data wrangling before making models in R. Of course, the J language is not for the faint of heart but it's really well-suited to the task.
I have just embarked on an adventure to do just what you describe in... Racket. But it's nowhere to be seen yet.
I'm an epidemiologist and I've been wanting to make my own tools for a while, now. It'll be interesting to see how far I can go with Racket, which already includes many pieces of the puzzle.