"much faster than Pickle"
That isn't saying much, though.
https://www.benfrederickson.com/images/python-serialization/...
Unrelated: What is this *pickle* term origin?
Python errors sometimes generate some "Could not pickle" errors and not sure what it tries to convey ...
Pickling a cucumber stores and preserves it for later use, so the term is used for serialization. Still an odd analogy though, since you can't unpickle a pickle. I've heard freeze/thaw which is at least a little better.
It's Python. So it's Monty Python.
Re:Apache Arrow 3.0
by bobbylarrybobby on Wednesday February 03, 2021 @09:56PM
(#26022057)
Pickle is the Python module that (de)serializes Python objects to some sort of binary format. Failure to pickle means the object couldn’t be serialized — the most common instance is probably trying to pass a lambda to a multiprocessing worker, because they don’t have names (you should use a def or functions.partial instead).
I have a question about whether it would fit this use-case:
* I need a SUPER fast KV-store.
* I'm on a single machine.
* Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as an 8-byte int. sqlite said it supports arbitrary precision numerics, but then I got burned finding out that casts integers to arbitrary precision floats and only keeps the first 14 digits of precision :\
* Values are 4-byte ints. Maybe 3 4-byte ints.
* I have maybe 10B - 100B rows.
* I need super fast lookup and depending upon my machine can't always cache this in memory, might need to work from disk.
Would arrow be useful for this? Currently just using sqlite.