News for hackers. Stuff that matters.
Re:SQLite small blob storage: 35% Faster Than the Filesystem by tetha on Wednesday June 14, 2017 @04:38AM (#14551411)
> Despite all this, my practical experience is that filesystem is better than blobs for things like uploaded content, images, pngs and jps etc.

This is especially true if you want to deliver them back to the users in the context of a web application. If you place this kind of content in a database, you'll need to serve them with your application.

If you use files for this content, you get two interesting options. For one, you can use any stock web server like nginx to serve these files - and nginx will outperform your application in this context. On top of that, it's easy to push this content onto a CDN in order to further cut the latency to the user.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by rkeene2 on Wednesday June 14, 2017 @04:18PM (#14555774)
Completely shameless plug here: My webserver[1] will outperform nginx for serving up static content like this.

It has many optimizations, like not opening the file its serving (which reduces the latency of walking/interacting with the VFS layer); almost no runtime allocations on the heap (minimal stack frames and sizes as well) which can be completely disabled at the expense of logging; high performance logging which does not interfere with processing requests; the normal stuff in a fast webserver (sendfile, keep-alive, minimal code paths between reading the user request and delivering the first byte).

[1] http://filed.rkeene.org/

Re:SQLite small blob storage: 35% Faster Than the Filesystem by tetha on Wednesday June 14, 2017 @05:20PM (#14556219)
Color me interested. How do I health-check this in a LB? What kind of caching headers do you give me out of the box?
Re:SQLite small blob storage: 35% Faster Than the Filesystem by rkeene2 on Wednesday June 14, 2017 @05:25PM (#14556242)
You can check its health by making a request to a known resource -- they're practically free.

It delivers an ETag header that it computes the first time it opens a file and maintains as long as that file is open. If it has to re-open the file it will generate a new ETag value.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by rusanu on Wednesday June 14, 2017 @05:06AM (#14551486)
Amen to that. Fastest database query is the one you never run.
Re:SQLite small blob storage: 35% Faster Than the Filesystem by icebraining on Wednesday June 14, 2017 @05:38AM (#14551589)
If you place this kind of content in a database, you'll need to serve them with your application.

Well, not necessarily: https://github.com/FRiCKLE/ngx_postgres/

Re:SQLite small blob storage: 35% Faster Than the Filesystem by rusanu on Wednesday June 14, 2017 @07:20AM (#14551935)
In this case nginx is the 'application'. the requests is still going to be expressed as a SQL query, sent to the PG, parsed, compiled, optimized, executed, then the tabular response formatted as the HTTP response. Many more steps compared to a file-on-disk response.

But I second that is an interesting nginx module

Re:SQLite small blob storage: 35% Faster Than the Filesystem by icebraining on Wednesday June 14, 2017 @07:47AM (#14552027)
Ah, but the filesystem has to do all that as well! It must receive a path, parse it, and then execute the query, with possible optimizations (eg. ext4 even has indexes implemented with hashed b-trees).

A filesystem is just an hierarchical database.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by cat199 on Wednesday June 14, 2017 @10:15AM (#14552861)
Does your database transparently DMA the buffer cache to the NIC after loading the file in the background using read-ahead without any process context switching or upcalls to userland?

That's what I thought.

https://www.freebsd.org/cgi/man.cgi?query=sendfile&sektion=2

http://man7.org/linux/man-pages/man2/sendfile.2.html

Re:SQLite small blob storage: 35% Faster Than the Filesystem by coldtea on Thursday June 15, 2017 @08:54AM (#14559902)
Does your filesystem provide for its files that are referenced by your DB the same versioning, consistent backups, and ACID guarantees as the DB does?

That's what I thought.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by fnord123 on Wednesday June 14, 2017 @09:14AM (#14552448)
>A filesystem is just an hierarchical database.

Loosely. Today's file systems aren't transactional (i.e. acid compliance), which is a basic property that most people consider necessary for a database.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by the8472 on Wednesday June 14, 2017 @01:46PM (#14554492)
Many filesystems are logged and provide atomic operations. You can build multi-step transactions on top of them with various operations. And with CoW filesystems you can even get read consistency. Oh, and optimistic locking also is an option.
Re:SQLite small blob storage: 35% Faster Than the Filesystem by fnord123 on Wednesday June 14, 2017 @06:07PM (#14556496)
Are we talking about the POSIX interface to files or is there a new transactional API that allows for atomic writes (all pages are written to disk or none are), consistency, isolation (opening a file in another process which is currently being written won't show the pages that are being written), and makes the writes always durable (sync; not fsync)?
Re:SQLite small blob storage: 35% Faster Than the Filesystem by the8472 on Wednesday June 14, 2017 @08:15PM (#14557033)
> Are we talking about the POSIX interface to files

Those give you these atomic ops and optimistic or cooperative locking: https://rcrowley.org/2010/01/06/things-unix-can-do-atomicall...

Multi-file transactions can be built by moving whole directories over symlinks to the previous version. The linux-specific RENAME_EXCHANGE flag can simplify this.

> or is there a new transactional API

CoW filesystems give you snapshots and reflink copies via ioctls. Under heavy concurrency this can provide cheaper isolation than locking.

> and makes the writes always durable (sync; not fsync)?

For durability fsync is sufficient if you do it on the file and the directory. To combine atomic and durable you can do the write, fsync, rename, fsync dir dance.

https://lwn.net/Articles/457667/

Re:SQLite small blob storage: 35% Faster Than the Filesystem by fnord123 on Thursday June 15, 2017 @11:03AM (#14561038)
>For durability fsync is sufficient if you do it on the file and the directory. To combine atomic and durable you can do the write, fsync, rename, fsync dir dance.

Right. This allows for the correct behaviour but I wouldn't conclude that POSIX is a transactional API. I would regard this as being able to build a transactional API on top of it; but not transactional in itself. I mean, if you had to do the same dance in SQL (insert to dummy pkey and then update the entry) you would rightly dismiss it.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by the8472 on Thursday June 15, 2017 @01:01PM (#14562162)
Which is why I said

> You can build multi-step transactions on top of them with various operations.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by phamilton on Wednesday June 14, 2017 @10:36AM (#14553017)
That's not really a true statement. I use plenty of databases that aren't transactional beyond a single record.
Re:SQLite small blob storage: 35% Faster Than the Filesystem by _jal on Wednesday June 14, 2017 @11:09AM (#14553291)
It is a true statement. You're not a member of the set of "most people":

> a basic property that most people consider necessary

I use non-ACID stores for some purposes, but don't consider them real databases.

Re:SQLite small blob storage: 35% Faster Than the Filesystem by justinclift on Wednesday June 14, 2017 @06:00AM (#14551644)
Interesting project. Although that one seems pretty dead (2 years since last commit), this fork of it seems actively developed:

https://github.com/konstruxi/ngx_postgres

Re:SQLite small blob storage: 35% Faster Than the Filesystem by sbov on Wednesday June 14, 2017 @12:33PM (#14553858)
It's pretty easy to use a CDN with blob storage, which reduces the impact of serving it from your application.