I have been using SQLite DBs to that end with some amount of success. If I remember well a lot of software of yore used BDB to the same effect.
Apache Arrow?
As far as I understand it, it's more for cross-process in-memory fast access (e.g. numbers crunching), but it's mutable.
As far as I can tell, Apache Arrow is not very mutable. It basically has to reallocate the universe.
I don't need large data between processes. I need short-lived medium data ... large enough and short-lived enough to make serialization painful but mutable enough to make "stores" kind of annoying.
Re:Parquet: More than just “Turbo CSV”
by manigandham on Sunday April 16, 2023 @06:37AM
(#35588814)
Use Redis. It's an in-memory data structure server, first created to have the same data structures usable and accessible across processes and languages.
Or plenty of other in-memory database products (Apache Ignite) or fast key/value (ScyllaDB) or just use RDBMS (Postgres or SQLite).
One thing that I have been struggling with is to find an in-memory, MUTABLE data format/data structure that can be used between different processes in shared memory. I'm not sure one exists.