Re:Comparing SQLite, DuckDB and Arrow with UN trade data
by tomnipotent on Wednesday October 27, 2021 @04:32AM
(#29010798)
Arrow eliminates ser/der, and if all actors in the workflow use the format you could see drastic performance improvements for a wide variety of workloads. I've seen ser/der for multi-GB+ processes take up half of the total clock time of the task.
Adoption is slow, but it'll get there.