Re:Comparing SQLite, DuckDB and Arrow with UN trade data
by twobitshifter on Wednesday October 27, 2021 @08:16AM
(#29012194)
My use case is that since Arrow keeps all data types and dumps the in memory table to disk, this allows me to backup my work and later reload the data and keep going. Loading and writing the data to disk is very fast It’s much better than using hdf5 for me in that regard.
[ Parent
| on HN ]
Re:Comparing SQLite, DuckDB and Arrow with UN trade data
by beagle3 on Wednesday October 27, 2021 @03:46PM
(#29018188)
And in most cases, if you memory map (mmap on Linux/BSD, MapViewOfFile on windows) it’s way faster than reading the file - because you only ever read what’s needed on one hand, and it stays in cache between invocations.