Re:Apache Arrow and MinIO
by mekster on Monday September 21, 2020 @12:43AM
(#24539885)
What is the use case of minio?
Typically server disks are much more expensive and turning it into s3 compatible storage doesn't seem like a good way of utilizing it.
[ Parent
| on HN ]
We’re using Minio storage on our own hardware to save on S3 egress transfer fees. It wouldn’t be very attractive based on the storage fee alone, but if you have any traffic at all, the traffic will dominate your calculations.
Of course, you can also get these savings with an online service like Wasabi or Backblaze B2. I just feel that using S3 as your only storage is fine, but once you move to the budget options you need an external backup. So we are using Minio and backing up to Wasabi, and still saving a lot.
I wonder but if you have access to the actual file system, why put the S3 layer unless the app is already built to use S3 in mind?
Re:Apache Arrow and MinIO
by stingraycharles on Monday September 21, 2020 @01:57AM
(#24540260)
We use minio in our on-prem dev and CI environments, seems to be perfect for this use case.
Re:Apache Arrow and MinIO
by selfhoster11 on Monday September 21, 2020 @05:52AM
(#24541520)
Non-commercial workloads that don't need 11 nines of reliability. S3 is fairly expensive for those use cases, so Minio is a good option to replace it if you need an object-based data store (for backups, for instance).
Why do you need S3 API layer for backup when you have the actual file system which is easy to provide sftp or rsync over openssh?
Re:Apache Arrow and MinIO
by selfhoster11 on Monday September 21, 2020 @05:36PM
(#24548749)
S3 is HTTPS-based, and all you need to implement it is access to HTTPS and HMAC hashing (for computing signatures). This allows it to be used with virtually any platform and programming language, because uploading a file is as simple as making a HTTP PUT request with a few extra headers set (assuming there isn't already an official or unoffical SDK that you can leverage instead).
S3 is considerably simpler than SFTP or rsync over SSH on non-Unix-like platforms, which makes it a very interoperable API. Even Wordpress or a small Go service can implement S3 support without much effort or any extra dependencies. Being HTTPS-based, it can also be reverse proxied more easily because HTTP presents a Host header, unlike SSH.
Re:Apache Arrow and MinIO
by MogwaiAllOnYou on Monday September 21, 2020 @04:53AM
(#24541182)
We use it as for on-prem deployments where cloud deployments use S3, also useful to spin up for local dev with Vagrant