Re:Mayastor: Lightning Fast Storage for Kubernetes
by fh973 on Sunday November 08, 2020 @03:52PM
(#25028907)
You are able to attach the block devices, but local file systems do not support being mounted on more than one host.
For that you need to use a cluster file system, but that's a different beast.
Re:Mayastor: Lightning Fast Storage for Kubernetes
by antonvs on Monday November 09, 2020 @04:38AM
(#25033044)
The multi-reader scenario definitely supports multiple readers of the same file system - we use it both in and out of Kubernetes on GCP.
Kubernetes has specific support for this with its ReadOnlyMany access mode: https://kubernetes.io/docs/concepts/storage/persistent-volum...
However, I do see on that page that AWS EBS doesn't support this mode, so perhaps that's what you're thinking of.
Re:Mayastor: Lightning Fast Storage for Kubernetes
by hardwaresofton on Monday November 09, 2020 @06:13AM
(#25033570)
Would you mind going into detail on your setup? What do you use to do this outside and inside GCP?
Re:Mayastor: Lightning Fast Storage for Kubernetes
by antonvs on Monday November 09, 2020 @09:44PM
(#25042366)
Both scenarios I was referring to are on GCP, but one is on k8s (GKE) and one uses ordinary GCP VMs. In both cases we're using the multi-reader mounting feature of GCP persistent disks, all that changes is how that's specified.
Outside GCP, we do have an on-premise product that can deploy on any k8s cluster. In that case, the default is for the customer to provide an NFS mount for the data in question. We use the same k8s ReadOnlyMany access type for that.
I don't know OpenEBS, but this isn't true of block storage systems in general. Both AWS and Google allow mounting virtual disks to multiple pods or nodes if all but one are mounted read-only.
Even multi-node writing is possible - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volu... :
> Amazon EBS Multi-Attach enables you to attach a single Provisioned IOPS SSD (io1) volume to up to 16 Nitro-based instances that are in the same Availability Zone. You can attach multiple Multi-Attach enabled volumes to an instance or set of instances. Each instance to which the volume is attached has full read and write permission to the shared volume.
Google offers this too: https://cloud.google.com/sdk/gcloud/reference/beta/compute/d...
Of course this does have constraints compare to a true distributed filesystem, which the AWS page describes.