Re:Introducing Kubeflow: Composable, Portable, Scalable ML S...
by jdoliner on Thursday December 21, 2017 @03:36PM
(#15982513)
I have a lot of ideas about this. I'm one of the creators of Pachyderm, which does data pipelining on Kubernetes. The first concrete thing that I think K8s could have would be some sort of a notion of a DataSet. Right now the closest thing I think is a gitRepo volume. GitRepo volumes are different from most other k8s volumes in that they're not meant for persisting state, they're meant as a way to inject data (which might actually be code) into a k8s environment. I'd like to see that idea expanded, I could easily imagine a similar thing that allowed you to mount data from Object stores like s3 and GCS, that would do a lot to make K8s immediately useful for data workflows. Being able to output to volumes like this and pass them between various pods would be the next logical step.
[ Parent
| on HN ]
Re:Introducing Kubeflow: Composable, Portable, Scalable ML S...
by TheIronYuppie on Thursday December 21, 2017 @05:16PM
(#15983354)
This is an amazing idea! Have you thought about exploring to see if we can overload CSI to accomplish something like this? https://github.com/kubernetes/community/blob/master/contribu...