News for hackers. Stuff that matters.
Re:SQLite Is Serverless by sly010 on Sunday January 26, 2020 @04:06PM (#22154797)
I have been using SQLite as a format to move data between steps in a complicated batch processing pipeline.

With the right pragmas it is both faster and more compact than JSON. It is also much more "human readable" than gigabytes of JSON.

I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

Re:SQLite Is Serverless by SQLite on Sunday January 26, 2020 @07:54PM (#22156010)
> I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

The sqlite3_deserialize() interface was created for this very purpose. https://www.sqlite.org/c3ref/deserialize.html

Re:SQLite Is Serverless by paulfurtado on Sunday January 26, 2020 @04:51PM (#22155030)
If the language's sqlite bindings don't offer a way to load a database from a string, if you're on a modern linux kernel (3.17+) you can make use of the memfd_create syscall: it creates an anonymous memory-backed file descriptor equivalent to a tmpfs file, but no tmpfs filesystem needs to be mounted and there's no need to think about file paths.
Re:SQLite Is Serverless by palotasb on Sunday January 26, 2020 @04:24PM (#22154893)
You can use the memvfs module to load in-memory databases if you're using the C API. I'm not sure how many higher-level APIs support it though.

[1] https://stackoverflow.com/a/53453338/3063 [2] https://www.sqlite.org/loadext.html#example_extensions [3] https://www.sqlite.org/src/file/ext/misc/memvfs.c

Re:SQLite Is Serverless by rakoo on Sunday January 26, 2020 @06:31PM (#22155626)
A very interesting approach is sqltorrent (https://github.com/bittorrent/sqltorrent): the sqlite file is shared in a torrent, and all queries will touch a specific part of the file, which is downloaded on-demand.

Also check https://github.com/lmatteis/torrent-net

Re:SQLite Is Serverless by kevas on Sunday January 26, 2020 @10:42PM (#22156711)
Incredibly odd, but so awesome
Re:SQLite Is Serverless by 0xbadcafebee on Sunday January 26, 2020 @10:01PM (#22156570)

  $ mount -t tmpfs none /some/path
  $ write db.sqlite /some/path/db.sqlite
  $ read db.sqlite

We've been abusing tmpfs for more than 10 years to get around the IO layer's failings. It's probably still a valid pattern.
Re:SQLite Is Serverless by FridgeSeal on Sunday January 26, 2020 @06:20PM (#22155563)
This is a amazing, I think you may have just solved and headed-off a huge number odd problems for me.

Could you talk more about what Pragmas you’ve been using and why?

Re:SQLite Is Serverless by jjgreen on Sunday January 26, 2020 @06:45PM (#22155704)
Not the OP, but I find `PRAGMA synchronous = OFF` makes the creation of DBs vastly faster ...
Re:SQLite Is Serverless by fctorial on Sunday January 26, 2020 @08:23PM (#22156148)
> I only wish there was a way to open an http-fetched SQLite database from memory so I don't have to write it to disk first.

Ramfs?

Re:SQLite Is Serverless by dredmorbius on Monday January 27, 2020 @04:57AM (#22158075)
tmpfs is the better-behaved option should you run out of resources, see:

https://www.jamescoyle.net/knowledge/951-the-difference-betw...

I'm still remembering old-school ramdisks under Linux which were finite in both number and size, both to quite small extents. I think there were 8 (or 12 or 16?) total ramdisks available, of only 2-4 MB each, configurable with LILO boot options.

That's now ... mostly taking up valuable storage in my own brain for no useful effect.

Re:SQLite Is Serverless by fctorial on Monday January 27, 2020 @07:58AM (#22158832)
It looks like a good intro, thanks. I wasn't aware of these technologies, but I knew it was possible to build an FS in RAM. So I just put these two keywords together.
Re:SQLite Is Serverless by dredmorbius on Monday January 27, 2020 @06:09PM (#22164831)
FWIW, I learned a few things researching my answer.

(A prime validation for answering questions, BTW.)

My first read was that the old-school ramfs / ramdisk limitations still held. I can't actually even find documentation on them, though I'm pretty sure I'm not dreaming this.

Circa 2.0 kernal IIRC, possibly earlier.

OK, some traces remain, see:

https://www.tldp.org/HOWTO/Bootdisk-HOWTO/x1143.html

Note that this is OBSOLETE information.

Re:SQLite Is Serverless by DelightOne on Sunday January 26, 2020 @05:34PM (#22155308)
What pragmas do you use? It sounds amazing!