News for hackers. Stuff that matters.
Re:SQLite small blob storage: 35% Faster Than the Filesystem by inetknght on Wednesday June 14, 2017 @12:38AM (#14550683)
> Writing your own methods to retrieve arbitrary chunks of data from a monolithic file would be a lot of work.

Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database.

(it really depends on what language and/or library(ies) you use and for what purpose)

Re:SQLite small blob storage: 35% Faster Than the Filesystem by imron on Wednesday June 14, 2017 @01:49AM (#14550867)
> Oh goodness have we come a long way if interacting with a file is harder than interacting with a SQL database.

It's not interacting with a file, it's interacting with a file that contains other files of varying sizes that need to be accessed randomly with good performance.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by diroussel on Wednesday June 14, 2017 @03:21AM (#14551183)
If you only want to read then both zip and tar are good for this purpose. It's commonly used and known to work well. There are many formats that are really just a renamed zip file. Like jar, and epub. File entries don't have to be compressed so you can mmap.

If you want to do random writes then more consideration is needed.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by imron on Wednesday June 14, 2017 @03:57AM (#14551287)
> If you only want to read then both zip and tar are good for this purpose

And programmatically accessing them is significantly more complicated than using SQLite which involves dropping a single header in your project and about 10 lines of code.

> If you want to do random writes then more consideration is needed.

Which you almost certainly what you want to do for the sorts of use cases where SQLite is also under consideration.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by dkersten on Wednesday June 14, 2017 @05:14AM (#14551509)
which involves dropping a single header in your project and about 10 lines of code.

10 lines of user code + the sqlite library. Using other formats like zip is hardly any more user code (+ some external library).

Re:SQLite small blob storage: 35% Faster Than the Filesystem by imron on Wednesday June 14, 2017 @05:30AM (#14551554)
The sqlite 'library' is a single header file and a single c file that you can just drop in your project and use.
Re:SQLite small blob storage: 35% Faster Than the Filesystem by dkersten on Wednesday June 14, 2017 @10:29AM (#14552960)
So are many libraries for other formats.

Btw I'm personally a big fan of SQLite, I just think the argument here is a bit of a straw man.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by akx on Thursday June 15, 2017 @10:10AM (#14560539)
Afaik, tar files do not contain a central directory, so finding a certain file in a tar file requires skimming through the entire archive.
Re:SQLite small blob storage: 35% Faster Than the Filesystem by function_seven on Wednesday June 14, 2017 @01:01AM (#14550740)
As far as I know, querying a database has always been easier than querying a custom file format. It's the whole reason databases exist in the first place. And using one off the shelf is clearly easier than writing your own.
Re:SQLite small blob storage: 35% Faster Than the Filesystem by tetromino_ on Wednesday June 14, 2017 @01:18AM (#14550792)
Not easier, but vastly more flexible.

Sure, it's completely trivial to read a file containing an array of C structs of metadata and offsets into a blob of concatenated PNGs: just mmap the thing, and your main data structure is sitting right there in memory.

But problems start arising when you need to update the structure of that metadata. Add some more fields. Some more structured, multi-value fields, referring to multiple entities in your increasingly complex data structure. Entities which must be guaranteed to exist. And perform modifications concurrently, from multiple processes. And have the file not get corrupted when something kills a writer process in the middle of writing...

Databases are slow and overkill for the simple case, but they certainly make adding unexpected requested features much easier. And it can be expected that you will get unexpected feature requests.