News for hackers. Stuff that matters.
Re:SQLite as an Application File Format (2014) by ramraj07 on Monday December 24, 2018 @08:34PM (#18755335)
What if I need to store binary data like images or PDFs? Is storing them as blobs an acceptable method?
Re:SQLite as an Application File Format (2014) by anonymouzz on Monday December 24, 2018 @08:44PM (#18755362)
Yup, see no reason why not. This is answered in the post:

> Any application state that can be recorded in a pile-of-files can also be recorded in an SQLite database with a simple key/value schema like this:

   CREATE TABLE files(filename TEXT PRIMARY KEY, content BLOB);
> If the content is compressed, then such an SQLite Archive database is the same size (±1%) as an equivalent ZIP archive, and it has the advantage of being able to update individual "files" without rewriting the entire document.

Also look at the related post which suggests doing precisely that - storing images inside SQLite blobs: https://www.sqlite.org/affcase1.html

Re:SQLite as an Application File Format (2014) by nodamage on Monday December 24, 2018 @11:27PM (#18755893)
Is this really a good idea though? I thought it was generally preferred practice to store large binary files outside of the database. Otherwise the database file will end up growing pretty quickly.
Re:SQLite as an Application File Format (2014) by manigandham on Tuesday December 25, 2018 @12:57AM (#18756175)
That's your call. If you're going to be storing 100gb of images then put them somewhere else, but this post is about using SQLite as a file format so storing binary files inside is not a problem.
Re:SQLite as an Application File Format (2014) by out_of_protocol on Tuesday December 25, 2018 @03:08AM (#18756514)
Counterpoint: tiled maps, which are basically loads and loads of images, can be stored on disk as

- Direcory structure

- Zip file

- Tar file

- Sqlite db

Out of these, sqlite is the most compact, by far (also the fastest)

Re:SQLite as an Application File Format (2014) by fatboy on Tuesday December 25, 2018 @10:04AM (#18757647)
We just transitioned our tiled images from using a directory structure to SQLite.

Having a single file per very large image (for mobile) makes transferring them to the phone over USB around ten times as fast. I wish I’d done it years ago.

Re:SQLite as an Application File Format (2014) by marcosdumay on Tuesday December 25, 2018 @01:03PM (#18758394)
Creating huge files is usually a bad idea. It does not matter if they are a sequence of lines of characters, a gziped collecion of XML data, a binary format that requires 6000 pages of documentation, or a SQLite database.

If you can spread your data among many files, you probably should. Not everybody can.

Re:SQLite as an Application File Format (2014) by jamougha on Tuesday December 25, 2018 @05:45AM (#18756906)
Storing large objects in database rows impacts performance and scalability but this shouldn't be a problem unless your files are extraordinarily large.
Re:SQLite as an Application File Format (2014) by wongarsu on Monday December 24, 2018 @10:59PM (#18755824)
Yes. The MBTiles Format [1][2] is essentially just an SQLite database storing map tiles (thousands of square images of parts of a map). In that case it's mostly done because SQLite stores small files much more efficiently than a general purpose file system in default settings.

1: https://github.com/mapbox/mbtiles-spec 2: https://wiki.openstreetmap.org/wiki/MBTiles

Re:SQLite as an Application File Format (2014) by combatentropy on Monday December 24, 2018 @10:30PM (#18755720)
In SQLite, a database file can be 140 terabytes, https://www.sqlite.org/limits.html

And reading and writing can be even faster than if they were stored directly in the filesystem, https://www.sqlite.org/fasterthanfs.html

Re:SQLite as an Application File Format (2014) by devmunchies on Monday December 24, 2018 @11:09PM (#18755845)
So does that mean serving static files from a SQLite file would be faster than something like NGINX?
Re:SQLite as an Application File Format (2014) by oscargrouch on Monday December 24, 2018 @11:47PM (#18755960)
No, because of the sendfile syscall, where the kernel pass all the file data without switching back to userspace.

Even if you can in theory save some time with small files with sqlite, you would be back to read/write IO and userspace/kernel byte buffer switching.

And it probably wouldn't be faster than something like a mmaped file anyway (unless the SQLite Db is also mmaped).

Re:SQLite as an Application File Format (2014) by fiddlerwoaroof on Tuesday December 25, 2018 @12:36AM (#18756107)
The thing is SQLite handles a lot of hard problems that you might not realize you have: like “was my data actually written to the disk”
Re:SQLite as an Application File Format (2014) by qwerty456127 on Tuesday December 25, 2018 @03:05AM (#18756506)
Why don't we replace entire filesystems with SQLite then?
Re:SQLite as an Application File Format (2014) by darkpuma on Tuesday December 25, 2018 @04:23AM (#18756705)
>"This file contains some example code demonstrating how the SQLite vfs feature can be used to have SQLite operate directly on an embedded media, without using an intermediate file system."

https://www.sqlite.org/src/doc/trunk/src/test_onefile.c

https://www.sqlite.org/vfs.html

IIRC, during one of his talks D. Richard Hipp mentions this actually being done for real.

Re:SQLite as an Application File Format (2014) by phiresky on Tuesday December 25, 2018 @09:31AM (#18757546)
SQLite does not support BLOBs larger than 2GB, so in that sense it's only as good as FAT32