News for hackers. Stuff that matters.
Re:File Attachments: Databases can now store files and images by ltbarcly3 on Wednesday August 30, 2023 @01:51PM (#37326058)
I think there is too much emphasis on 'the right way' sometimes, which leads people to skip over an analysis for the problem they are solving right now.

For example, if you are building something that will have thousands or millions of records, and storing small binaries in postgresql lets you avoid integrating with S3 at all, then you should seriously consider doing it. The simplification gain almost certainly pays for any 'badness' and then some.

If you are working on something that you hope to scale to millions of users then you should just bite the bullet and integrate with S3 or something, because you will use too much db capacity to store binaries (assuming they aren't like 50 bytes each or something remarkably small) and that will force you to scale-out the database far before you would otherwise be forced to do so.

Re:File Attachments: Databases can now store files and images by crooked-v on Wednesday August 30, 2023 @02:12PM (#37326427)
> and that will force you to scale-out the database far before you would otherwise be forced to do so

Disk space is cheap, even when it's on a DB server and not S3. Why worry that much about it?

Re:File Attachments: Databases can now store files and images by tomnipotent on Wednesday August 30, 2023 @02:26PM (#37326656)
> Disk space is cheap

Disk I/O less so. An average RDBMS writing a 10MB BLOB is actually writing at minimum 20-30MB to disk - once to journal/WAL, once to table storage, and for updates/deletes a third copy in the redo/undo log.

You also get a less efficient buffer manager with a higher eviction rate, which can further exasperate disk I/O.

Re:File Attachments: Databases can now store files and images by crooked-v on Wednesday August 30, 2023 @02:32PM (#37326735)
But if the files are important enough that you need transactional updates or point-in-time recovery for them, you're stuck with the equivalents of those things anyway, wherever those files end up, plus the extra overhead of coordinating two systems.
Re:File Attachments: Databases can now store files and images by ltbarcly3 on Thursday August 31, 2023 @07:50AM (#37335746)
But there is a hard limit to db scale up and you don't want to hit it. Also, who said you need transactional updates or pit recovery in all cases?
Re:File Attachments: Databases can now store files and images by ransom1538 on Wednesday August 30, 2023 @03:40PM (#37327843)
Disk space is cheap. Slap on a db, now you got cpu, memory + an entire db OS inside an OS. Now its an expensive mess.
Re:File Attachments: Databases can now store files and images by sorintm on Wednesday August 30, 2023 @02:55PM (#37327112)
Brilliant point! It always depends on the problem you're trying to solve. What Xata did, was integrate files as a database column, BUT stored the binary content in S3. Thus offering both transactional file management and point in time recovery as well as high performance direct (S3) file access, through a CDN. Basically Xata implements the orchestration and takes away the pain of managing 2 services.
Re:File Attachments: Databases can now store files and images by ltbarcly3 on Thursday August 31, 2023 @07:52AM (#37335762)
The last thing I would do is use a proprietary solution from a startup to store data. Sorry, but that is just absolutely insane.
Re:File Attachments: Databases can now store files and images by harshitaneja on Wednesday August 30, 2023 @05:36PM (#37329458)
Going on a tangent here but why s3? Why not avoid vendor lock in and just serve static files?

My setup is to post blobs over a small rest api which balances storage load over all the file servers using nginx. It responds with the serverSubdomainId which in turn gets saved in the database. So the url for file access can be generated https://{serverSubdomainId}.domain.tld/{resourceId} The only costly part is writing which with caching of available storage of all the servers isn’t bad. The whole system is horizontally and vertically scalable. What am I missing that s3 is still the defacto even with vendor lock in and high egress costs?

Re:File Attachments: Databases can now store files and images by aforwardslash on Wednesday August 30, 2023 @06:18PM (#37329899)
There are plenty of oss solutions that talk "s3", like swift, ceph and seaweedfs.

Why object storage? Its easier to backup, version and to scale horizontally. Most solutions will also provide encryption capabilities that can be commanded either via external key management systems or from the client application. Also, custom access policies are great for private documents and file uploads.

Using static files is a good solution in some cases, nothing against it. But in many scenarios, there are huge benefits on using object storage instead, even if it is quite slower.

Re:File Attachments: Databases can now store files and images by timando on Wednesday August 30, 2023 @06:41PM (#37330094)
"S3" is often used to refer to any S3 compatible object storage system. You can even self-host it with something like MinIO.
Re:File Attachments: Databases can now store files and images by connordoner on Wednesday August 30, 2023 @07:00PM (#37330243)
I can't speak for everyone but, in my case, it comes down to the ease of setup and having someone else manage it.