News for hackers. Stuff that matters.
Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by Jean-Papoulos on Wednesday June 12, 2024 @07:52AM (#40657008)
Strangely enough I'm doing the exact same thing. My case seems a bit more complicated since we're dumping not only long arrays of floats, but also a bunch of miscellaneous data (datetimes, unique datapoints, ids) right before/after those numbers. So it's even more of a nightmare... Some of these files have version numbers in the 30's (oh yeah, and file versions are based on file size.)

And did I mention that the file handles are kept open for the whole program duration (might be multiple hours) ? That's fun, data loss is almost a daily occurrence. It's the main thing I'm refactoring right now.

Re:Why SQLite Is Taking over with Brian Holt and Marco Bambini by romwell on Wednesday June 12, 2024 @05:04PM (#40663101)
Well, with an ORM you put everything in data structures, and write them.

Most data fits into hierarchical data structures that don't form a complicated memory graph with references.

Once you know how to write primitive types and arrays of them, you can write objects, and it means you can write anything (without having to worry about writing "before" or "after").

And yes, switching to SQLite absolutely addresses the "perpetual fopen” problem.