News for hackers. Stuff that matters.
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by Lunatic666 on Tuesday March 20, 2018 @06:50AM (#16627370)
Is there any open source S3 compatible software? I know Riak Cloud Storage (http://docs.basho.com/riak/cs/2.1.1/), but I think it’s not maintained anymore.
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by notacoward on Tuesday March 20, 2018 @08:06AM (#16627769)
Minio and Ceph both fit this description, with the latter also offering block and file access. Swift is close, but a slightly different native API (there's an adapter but I'm not sure of its status). Scality has open-sourced their implementation as Zenko. Also LeoFS (which despite the name is not a filesystem).
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by gizzlon on Tuesday March 20, 2018 @07:02AM (#16627440)
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by cyberferret on Tuesday March 20, 2018 @06:52AM (#16627381)
I believe minio.io provides open source S3 compatible storage on your own servers.
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by unixhero on Tuesday March 20, 2018 @06:56AM (#16627400)
True
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by retrack on Tuesday March 20, 2018 @09:39AM (#16628415)
There is pithos [1] from the Exoscale [2] and powers the production system. A new version of the open source project with API signature v4 and other updates is due out soon.

[1] http://pithos.io/ [2] https://www.exoscale.com/object-storage/

Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by doublerebel on Tuesday March 20, 2018 @10:42AM (#16628970)
Joyent's Manta has an S3 adapter, all open-source.
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by tobias3 on Tuesday March 20, 2018 @06:57AM (#16627412)
OpenStack Swift or Ceph with Ceph Object Gateway. Don't use minio, it's a toy for testing.
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by notacoward on Tuesday March 20, 2018 @07:30AM (#16627574)
That's libelously untrue. You might think Ceph or Swift are better, that's fine, but it's no toy. I've seen a few toy S3 implementations. I even called out Zettar on my blog back in the day. If it's not scalable or highly available it's a toy. If it's unstable it's a toy. If it's super-duper slow it's a toy. Minio is none of these things, plus it has features like erasure coding and encryption that are mature enough to be backed by real support. Best ever? I'm not saying that, but it's no toy.

Disclaimer: I have no affiliation with Minio. In fact, they're very nearly a competitor with the project I work on. However, Minio is full of ex-Gluster people and I consider many of them friends.

Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by striking on Tuesday March 20, 2018 @07:06AM (#16627457)
What's wrong with minio? It does its job and does it well. I did backups with it once upon a time, in production, and it never failed.
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by icebraining on Tuesday March 20, 2018 @07:03AM (#16627447)
I've never used it in production; in what ways is minio lacking compared to Swift and Ceph?
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by tobias3 on Tuesday March 20, 2018 @01:20PM (#16630562)
S3 is an abstraction that gives you a limited amount of operations with limited guarantees (e.g. eventual consistency, can only replace whole files). What you get in return is easier scalability and performance. If you use an S3 API to store files (like minio does) you give up power and gain nothing. So you are better off using NFS, samba, webdav, ftp, etc. Additionally minio doesn't seem to sync files to the file system, so you can't be sure a file is actually stored after a PUT operation (AWS S3 and swift have eventual consistency and Ceph has stronger guarantees).
Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by icebraining on Tuesday March 20, 2018 @06:42PM (#16633479)
That's curious, because the OpenStack Swift docs say "Objects are stored as binary files on the filesystem with metadata stored in the file’s extended attributes (xattrs)".

As far as I know, the advantage comes from exposing a course-grained API over the network, which is generally more efficient and reliable than doing many small operations. It would be hard to implement Minio's distributed mode using the filesystem API.

Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by unixhero on Tuesday March 20, 2018 @06:56AM (#16627407)
The rclone project supports S3

https://rclone.org/s3/

Re:Object Storage: AWS vs Google Cloud Storage vs Azure Stor... by chrisper on Tuesday March 20, 2018 @07:01AM (#16627430)
Rclone.org