News for hackers. Stuff that matters.
Re:Why Is Storage on Kubernetes So Hard? by nemothekid on Sunday January 13, 2019 @06:37PM (#18899444)
AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc.

The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potentially anywhere. Once you have that, then storage on k8s is "easy".

Re:Why Is Storage on Kubernetes So Hard? by candiodari on Sunday January 13, 2019 @08:05PM (#18899808)
Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications.

Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT:

1) It is per-cluster, not global (given Google's cluster sizes one imagine's that's not that much of a limitation, but then again, it seems like that is a clear limit)

2) you can only create or append to files (or delete them I guess) (and this is therefore non-posix, and does not support things like mysql or postgres)

3) this is very different from what GlusterFS, NFS, persistent volumes, etc provide. Therefore disks on google cloud are presumably very much not just files on this GFS/colossus thing.

4b) it was single master at least until 2004. Maybe until 2010. Apparently that can work.

https://cloud.google.com/files/storage_architecture_and_chal...

https://static.googleusercontent.com/media/research.google.c...

Re:Why Is Storage on Kubernetes So Hard? by philsnow on Sunday January 13, 2019 @08:10PM (#18899829)
> GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications.

For the most part, google only has custom applications. Just about everything is written in-house, and takes advantage of things like being able to open a file from the local disk file just as easily as opening one in Colossus or their equivalent of Zookeeper.

Re:Why Is Storage on Kubernetes So Hard? by steelframe on Sunday January 13, 2019 @11:40PM (#18900698)
Would regional Persistent Disks be acceptable?

https://cloud.google.com/compute/docs/disks/#repds

Global might mean you're accessing blocks stored in Europe from a container running in Australia. If your workload is doing that, you might want to consider network costs and latency, and then reconsider whether you really want to architect things that way.

Re:Why Is Storage on Kubernetes So Hard? by nemothekid on Monday January 14, 2019 @02:02PM (#18905414)
You are entirely correct, Colossus isn't POSIX. However, the solution Google has, is since they control all the software, essentially everything they write acts as if Colossus is their storage layer.

This "solves" the persistent storage layer in a way Kubernetes cannot. Inside Google you might deploy your "Spanner" database container which knows how to interface with Colossus and doesn't require any special setup or colocation (below the cluster level). You cannot deploy MySQL on K8s and expect the same.