I though, one of the advantage here is, you are able to store/read metadata along the file content easy without worrying the metadata and file may desync in some way or corrupted. This makes serve something like Image (photo dimension) or video(length) metadata way easier. Some filesystem (like ntfs) do have place to save metadata along files, but most filesystem don't have the way to done that properly

You're referring to extended attributes, which are like a key/value database for each file. Wikipedia says:

> In Linux, the ext2, ext3, ext4, JFS, Squashfs, Yaffs2, ReiserFS, Reiser4, XFS, Btrfs, OrangeFS, Lustre, OCFS2 1.6, ZFS, and F2FS[9] filesystems support extended attributes (abbreviated xattr) when enabled in the kernel configuration.

> The Linux kernel allows extended attribute to have names of up to 255 bytes and values of up to 64KiB,[13] as do XFS and ReiserFS, but ext2/3/4 and btrfs impose much smaller limits, requiring all the attributes (names and values) of one file to fit in one "filesystem block" (usually 4 KiB).

Applications that make extensive use of xattrs typically recommend one of the FSs that allow full-size xattrs. For example, OpenStack Swift, an object storage service, recommends XFS as a backing storage. (Source: I operate Swift clusters at $dayjob.) But if your metadata is not larger than 4 KiB, any production-quality Linux FS will do.

Then you would encounter another problem, extended attributes arn't usually really portable between files system / operation systems (as the name 'extendded' suggests, it may vary between FSs or even not available at all). And may gone easily if any of the programs not operate on them correctly. Windows would show a big warning if your copy operation may cause the extra metadata along the file to gone, but I am not sure whether linux will do that.

Just store the metadata in the file name. Unless your file system besides metadata also have a 11 character filename limit.

What kind of versioning are your referring to? I know some databases has it but not sure if I understand it.

Personally I use S3 or similar for this. At the number of files I'm working with storing it in a relational database would be crazy expensive.

Scheme could be something like;

FileId, VersionId, Name, Data, UploadedTimestamp, UploadedBy, ...

If you’re on AWS, pushing and pulling from an S3 bucket is probably a great solution, and then of course there’s nothing to worry about in terms of backups.

Do you still need to keep an index of the files / metadata in a DB, or can you tag everything you need directly on the S3 objects and just pull the whole bucket?

Even if you’re not in AWS S3 works great, especially where you need to serve content direct to client’s browsers. S3 supports either making a file public and available to all, or using pre-signed URLs to provide short term authorisation to access a file to specific clients. No need to pull the bucket at all.

You will hit limits to how fast you can generate those crypto signatures for s3, and a limit to how much can serve; at a certain scale you will want to use a cdn

> You will hit limits to how fast you can generate those crypto signatures for s3

Presigned URLs are HMAC-SHA256:

https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-heade...

I just measured GetObjectRequest.Presign() at 1063 ns. That's a million signatures per second of my 2015 MacBook Pro using the stock AWS SDK for Go _on one core_. This is plenty fast already, and it's guaranteed to be running in parallel on better hardware in production. There's no way signature generation is a bottleneck in any real application.

> and a limit to how much can serve

I've gotten hundreds of gigabits per second between S3 and EC2 without doing anything special. What problems are you referring to?

> at a certain scale you will want to use a cdn

So use a CDN too. CloudFront fits there neatly, and it can even sign its own requests to S3:

https://aws.amazon.com/blogs/networking-and-content-delivery...

None of those criticisms are valid reasons to avoid S3. S3 is an excellent solution for many common problems.

Having said that, S3 has genuine drawbacks for certain workloads. (For example, the long tail on time to first byte is sometimes a dealbreaker.) In my opinion the biggest limitation is budgetary: typically one will spend all their money on AWS data transfer fees long before hitting S3 service limits.

Tons of tiny files is the problem where the bytes to sign ratio is low.

And I didn't say to avoid s3. I work for a subsiary of AWS I want you to use it a ton.

Let's say your serving up a bunch of thumbnails, maybe 1024 per user per load.

Or in my scenario 25k files. 25k signs per user is 40 users per second per server signing constantly.

For the last 6-8 months I've been working on a hybrid electron + web app that uses files on local disk for storage.

I have a Datastore implementation that uses disk when locally and firebase when remote.

https://getpolarized.io/2019/03/22/portable-datastores-and-p...

You can read about it there.

I thought it would be a great idea to use just simple flat files as you can commit them to git, diff them, and there's kind of already an API for working with them.

In retrospect I regret it and wish we had something more powerful.

First. It's twice as much code to maintain. Which introduces bugs.

Seconds if you're keeping your PDFs online vs on disk they're duplicated. It would be nice to build in a higher level abstraction that was both network AND disk aware and just pulled things down when you needed it.

Third, the annotations on your PDF aren't really indexable without a 3rd party system.

Getting the right balance of remote-first vs web-first is tough.

Now, after building a complex webapp with electron support I think it's better to go web-first and then try to use the browsers local caching to get offline-first support.

I once did this specifically to get the files into the database backups, so i didn't have to deal with them separately!

The files were the storage for some third-party content targeting tool, and were edited using an admin UI that tool, then deployed to the servers where the live part of the tool would read them. You couldn't bring the system up in a complete state without them.

Apart from those files, we could restore the whole system using the latest code (really, the artifact built from it), and the latest database dump. So, before taking a database dump, we'd run a script to load those files into the database. We also had a script for undumping them, which we tested regularly in our staging environment.

We didn't use the database for deployment, because the third-party tool did that. Perhaps we should have - it would have been a bit of a subversion of the tool's usual mode of operation, but it might have had some advantages.

>Really the only downside I ever found — at the scale I was operating at — was that it bloats the database backups.

Easily solved by having two separate databases. One for dynamic content, one for static files, which is probably good practice regardless.

Doesn't that introduce the desync problem again? Is there something that can enforce consistency across the two databases?

I'm assuming the static and dynamic content don't need to be transactionally integrated. If they do, you could still solve it with two databases, but you'd use one database for the "current" data and another database to store "historical" data.

Also, remember the GP said "small to mid-size."

Backups are fundamentally limited to eventual consistent, there is no need for databases to be synchronously replicated for backups. I mean splitting database has no effect on backup consistency, although a more decent way of dealing with it is not splitting database, but simply running an async replica to do backups from.

Are you are talking past each other...? I suspect the grandparent post is concerned about the loss of ACID properties for updating data and metadata if they are split across two databases. This is a concern for regular application access too. Meanwhile, you assume that this consistency problem is already solved in the application, and so of course the backup and restoration problem can also then be handled.

To the grandparent: you typically solve this application consistency issue by using something like immutable object store semantics with versioned references. The object store is capable of answering requests for multiple versions of the asset, and the application metadata store tracks individual versions. You can sequence the order in which you commit to these stores, so the asset is always available before it is published in the metadata store. Alternatively, you can make consumers aware of temporary unavailability, so they can wait for the asset to become available even if they stumble on the new version reference before the content is committed.

You can also find hybrids where the metadata store is used to track the asset lifecycle, exposing previous and next asset version references in a sort of two-phase commit protocol at the application level. This can allow consuming applications to choose whether they wait for the latest or move forward with older data. It also makes it easier to handle failure/recovery procedures such as canceling an update that is taking too long and reclaiming resources.

Backups on databases like PostgreSQL are strongly consistent, regardless of where you take it from (master or replica). Postgres replication is strictly sequential wrt transaction commit order.

As such, splitting the database may incur in significant consistency issues that a backup doesn't incur into.

I believe this splitting technique is not a good one except for potentially narrow use cases.

Strong consistency is not what you think then. You can only do stale reads from backups.

I thought ahachete meant that a database backup is atomic, i e. it will only contain fully completed transactions. The problem with data split across databases then is that transactions don't span multiple databases, so you can't get an atomic snapshot of data spanning multiple databases.

Are you saying that even a single-database backup is not atomic?

No, it's atomic. It's just not important if separate databases are not atomic to each other when you recover from backups, you will still have data loss and an inconsistent state. I.e. things that should be in the database missing, and that shouldn't be overwriting missing IDs, etc. Backups cannot be strongly consistent, so you have to take the exact same approaches to deal with this whether you store everything in a single database or in two separate ones.

My (naive?) view is that Nginx caches static files by default after the first hit so the inconvenience isn't worth it in my web apps. Would I really see an improvement switching to a SQLite-esque file system?

> My (naive?) view is that Nginx caches static files by default after the first hit

That's likely the OS's filesystem cache, not anything in nginx. If you're on Linux, you can confirm this by clearing the OS filesystem cache with

  $ sudo -i
  # echo 3 > /proc/sys/vm/drop_caches

and checking if the next hit on your URL looks like a cold load.

If the files are being handled programmatically, pull them as byte[] directly from the database.

Once I tried to hookup a way to serve the files directly from the DB through IIS, and it was a total failure. By the time I had figured out getting all the various headers right, I was still running into some weird caching issue that would corrupt the request at random intervals.

So now, if the files are static hosting assets, I will initialize a ‘cache/‘ folder on the local filesystem (for that machine on the web-farm) when the first request comes in and just read all those files back into the local file system for the web server to handle.

The only remaining concern is refreshing the cache when files change in the DB.

Nginx doesn't cache files by default, but it has a file based caching module and a file based store module that can be used to retrieve files from backends and cache on local filesystem.

> was that it bloats the database backups.

Having a deduplicating backup solution might help with that. Simplest way to do that (in terms of ease-of-implementation), would be to keep backups on a ZFS or btrfs filesystem. Note that (AFAIK) with btrfs, deduplication must be triggered manually, so you'd need to run `btrfs filesystem defragment` on the backups to take advantage of it.

Note for other readers: The parent post probably means incremental ZFS snapshots, which won't store multiple copies of any given block unless the piece of data has been moved.

ZFS also has a block deduplication scheme, but turning it on is a poor idea. The deduplication table needs to be stored in memory for decent performance, and this takes a lot of memory. Even then, performance is rarely as good as with dedup turned off. It doesn't even save as much space as you'd hope -- it's per-record, and ZFS storage records are 128KiB by default. If you're trying to deduplicate 10KiB files with that, you're going to have a bad time.

(Note also: The ZFS record size is dynamic; the 128KiB default is the maximum record size it'll allow, i.e. the most data that has to be read and rewritten (somewhere else) if you modify one byte in a larger file. If you just write a single 10KiB file and force it to disk, then you'll get a 10KiB record. It's complicated; try not to read more into this than "dedup is rarely a good idea," though if I'm allowed a second line, I'll add "compression is always a good idea; enable zle at a minimum.")

Have you seen https://icculus.org/physfs/ ?

You could put everything inside a zip and that's it.

Personal opinion: Those things (static assets) belong in a container with Nginx serving them (and that’s not even counting SAAS options like S3). Easy to integrate with CDNs, easy to update, easy to roll out and back. And you get to take advantage of layers to keep the network and disk usage minimal.

I don’t see how a container solves the backing store / source of truth problem?

How are changes getting put into the container? Are you versioning the container each time a file is changed or added? When files change are all instances of the containers being restarted to get the new/changed files?

One answer: the files are stored in git and the image is built on commit and tagged with the current ref, then pushed. The deployment is then updated to reference the new tag and services are restarted, pulling the new image and launching the new container. This is the typical ci/cd setup for kubernetes deployments.