News for hackers. Stuff that matters.

A comment on DTable a new distributed table implementation in Julia using Dagger.jl.

Re:DTable a new distributed table implementation in Julia us... by 1egg0myegg0 on Thursday December 09, 2021 @12:13AM (#29494065)
Have you thought about interfacing with DuckDB for out of core processing?

https://github.com/kimmolinna/DuckDB.jl

[ Parent | on HN ]
Re:DTable a new distributed table implementation in Julia us... by jpsamaroo on Saturday December 11, 2021 @09:15AM (#29520991)
Firstly, I'll say that we already have work started to implement out-of-core directly in Dagger: https://github.com/JuliaParallel/Dagger.jl/pull/289.

With that PR in place, it should be possible to define a "storage device" which is backed by a database. I haven't had a chance to actually try this, since the PR still needs quite some work and testing, but it's definitely something on my radar!

[ Parent | on HN ]