Re:BeeGFS, the Parallel Cluster File System
by hkothari on Wednesday February 24, 2016 @11:52AM
(#11168153)
Can someone explain how this differs from something like HDFS? I'm kind of confused as to why there's no comparisons or mentions of HDFS here which makes me think I'm missing something important about what this provides that's special. Is it the easy to install part?
Re:BeeGFS, the Parallel Cluster File System
by birc_a on Wednesday February 24, 2016 @12:25PM
(#11168433)
HDFS is designed mostly for immutable files, stores three copies of everything in the default setup and has the whole hadoop calculation stuff.
BeeGFS is closer to a normal NFS share. In the default setup it just stores the data once, striped over multiple machines and it supports efficient updates in-place etc.
Re:BeeGFS, the Parallel Cluster File System
by birc_a on Wednesday February 24, 2016 @12:03PM
(#11168238)
MooseFS and BeeGFS don't really seem to be in the same category of filesystem though.
BeeGFS comes from the HPC world where it is all about performance, while MooseFS seems more focused on high reliability even in the face of entire machines coming and going.
BeeGFS stores files striped over multiple machines and can use infiniband natively which gives us a system where bandwidth to individual files can reach a bit over 1GB/s (best case) and aggregated bandwidth can reach 30GB/s from a very mixed and unoptimized bioinformatics workload, a lot more when just testing the raw bandwidth.
Since BeeGFS uses an underlying filesystem on each storage target you can of course run raid, zfs or whatever it takes to make you comfortable that the individual storage targets aren't going to be lost - which is what it takes for data to be unavailable.
If you want some extra reliability in BeeGFS it also supports mirroring so you only lose data if you fully lose two storage targets. We can't really afford to run with full mirroring for our 3PB though.
We are very happy with it for our HPC environment but I'm not sure how well it works in an AWS setup.
Re:BeeGFS, the Parallel Cluster File System
by eis on Wednesday February 24, 2016 @07:40AM
(#11166255)
I'd be also interested in a bit of info why Ceph and GlusterFS disappointed you. I tried GlusterFS a few years ago and it took a good while to figure out the right setup but in the end had disappointing small file performance.
Ceph was very interesting but wasn't product ready when I looked at it and since then it somehow never got to a stage where people would rave about it so that alone makes me think it's maybe too complicated. A bit like BTRFS which also for years and years is being developed and sounds really promising but never seems to be finally ready without gotchas.
Re:BeeGFS, the Parallel Cluster File System
by mattbillenstein on Wednesday February 24, 2016 @09:11AM
(#11166797)
I also tried ceph and gluster before settling on moosefs a couple years ago -- gluster was slow for filesystem operations on a lot of files and it would get into a state where some files weren't replicated properly with seemingly no problems with the network for physical servers.
Ceph at that time was unstable -- once I had a couple boxes kernel panic in testing, I jumped off that boat immediately.
Moosefs is pragmatic -- written in pretty tight C, performant, and the web UI for seeing the state of the cluster and how files are replicating is very nice. We have a small cluster with maybe a dozen nodes running for almost 2 years now with no hiccups...
Re:BeeGFS, the Parallel Cluster File System
by mattbillenstein on Wednesday February 24, 2016 @09:56PM
(#11172056)
We were trying out the filesystem -- it was clearly alpha/beta quality at that time, so it was no surprise it didn't work, and perhaps it's improved in the last two years.
Re:BeeGFS, the Parallel Cluster File System
by scurvy on Thursday February 25, 2016 @12:20AM
(#11172538)
cephfs is clearly marked as not for production. It still has more than its fair share of bugs and has single points of failure. From the very top of the docs page on ceph FS:
"Important CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters"
RedHat is focusing on making the block and object storage bulletproof. They're content to let the opensource side twiddle away on ceph FS.
Re:BeeGFS, the Parallel Cluster File System
by shiftpgdn on Wednesday February 24, 2016 @02:17PM
(#11169367)
The company I work for has been running BeeGFS for 6 years with no issues. Write and read speeds on Dell storage hardware match or exceed the NetApp appliances we use.
We do occasionally have issues where ALL storage nodes must be rebooted but other than it works great.
Re:BeeGFS, the Parallel Cluster File System
by merb on Wednesday February 24, 2016 @09:20AM
(#11166866)
Actually there isn't a easy file system which spans 3 computers and scales to thousand and is eay to install.
There are only one's which are hard to install, or needing more than 3 computers, etc..
Re:BeeGFS, the Parallel Cluster File System
by angrygoat on Wednesday February 24, 2016 @10:26AM
(#11167397)
I've run a ceph deployment for about 18 months now, it's good. ops-wise it's like anything, a few quirks to learn - but it's very solid. I played around with a couple of puppet modules and ended up on ceph-deploy as well, makes life quite easy.
Re:BeeGFS, the Parallel Cluster File System
by Diederich on Wednesday February 24, 2016 @09:43AM
(#11167040)
Honest question: how does GlusterFS not meet these goals? I know that it's easy to install, and can easily run on three computers. And I believe it can scale to thousands.
Re:BeeGFS, the Parallel Cluster File System
by merb on Wednesday February 24, 2016 @11:22AM
(#11167890)
Actually GlusterFS is close to my needs, however the "easy to install" and actually bad performance on many small files is a road blocker for us. However it runs well, even on just 3 servers.
Re:BeeGFS, the Parallel Cluster File System
by craigyk on Wednesday February 24, 2016 @11:38AM
(#11168025)
I will say that I know two people that have independently tried fairly large trials (400TB+) of Lustre, GlusterFS, and BeeGFS, and BeeGFS was their eventual favorite.
Re:BeeGFS, the Parallel Cluster File System
by sharktour on Wednesday February 24, 2016 @08:40AM
(#11166586)
This doesn't fill the huge gap for Cloud NAS. GlusterFS is hugely complicated and a fragile install, SoftNAS has (anecdotally) poor performance on high-io... but it's closer.
Amazon hasn't launched their EFS product to General Availability Yet.
So wtf do we do when we need a 5TB shared storage platform across some EC2 nodes?
Re:BeeGFS, the Parallel Cluster File System
by illumin8 on Wednesday February 24, 2016 @12:07PM
(#11168281)
Check out Avere systems. They have a clustered NFS appliance that you can spin up in EC2, which translates back-end S3 object storage into NFS for your clients. It supports between 3 and 50 nodes, and uses local RAM and SSD cache to accelerate performance. Avere has also been in the NAS acceleration/caching business since 2008, so their product is very mature.
Re:BeeGFS, the Parallel Cluster File System
by sharktour on Thursday February 25, 2016 @08:17AM
(#11174147)
Thanks for the heads up on Avere. Use case for us was for file parsing as part of a batch data pipeline. The latency we saw on alternatives was killer compared to directly mounted EBS. That would be my fear with an S3 option as well, especially when it presents like NFS.
Re:BeeGFS, the Parallel Cluster File System
by Loic on Wednesday February 24, 2016 @07:03AM
(#11166077)
For what I read, this is to know a bit who is interested in getting the source code. I give you the code but you tell me who you are. This is not prohibited by the GPL.
Re:BeeGFS, the Parallel Cluster File System
by blinkingled on Wednesday February 24, 2016 @07:33AM
(#11166221)
Yeah, sounds like they would be better off putting up the sources on a public git repo and then develop it in the open. Closed FS development for Linux at least does not seem to work well based on history. They can always have paid support but the closed development part doesn't bode well.
Re:BeeGFS, the Parallel Cluster File System
by mixmastamyk on Wednesday February 24, 2016 @01:21PM
(#11168910)
It's a tragedy to think of how deep is your love for the project.
Wonder how long it will be stayin' alive? Gives me a night fever just thinking about it.
BeeGFS is closer to a normal NFS share. In the default setup it just stores the data once, striped over multiple machines and it supports efficient updates in-place etc.
Umm, how is this compatible with GPL v2? It seems more restrictive (enterprise features, parts that would be better enforced by a trademark policy).
https://indico.cern.ch/event/346931/session/4/contribution/6...
> Plain BeeGFS is as reliable as your hardware
currently i would give MooseFS, orangefs or lizardfs a try.
Had experience with ceph and glusterfs and both I can not recommend.
BeeGFS comes from the HPC world where it is all about performance, while MooseFS seems more focused on high reliability even in the face of entire machines coming and going.
BeeGFS stores files striped over multiple machines and can use infiniband natively which gives us a system where bandwidth to individual files can reach a bit over 1GB/s (best case) and aggregated bandwidth can reach 30GB/s from a very mixed and unoptimized bioinformatics workload, a lot more when just testing the raw bandwidth.
Since BeeGFS uses an underlying filesystem on each storage target you can of course run raid, zfs or whatever it takes to make you comfortable that the individual storage targets aren't going to be lost - which is what it takes for data to be unavailable.
If you want some extra reliability in BeeGFS it also supports mirroring so you only lose data if you fully lose two storage targets. We can't really afford to run with full mirroring for our 3PB though.
We are very happy with it for our HPC environment but I'm not sure how well it works in an AWS setup.
Ceph was very interesting but wasn't product ready when I looked at it and since then it somehow never got to a stage where people would rave about it so that alone makes me think it's maybe too complicated. A bit like BTRFS which also for years and years is being developed and sounds really promising but never seems to be finally ready without gotchas.
Ceph at that time was unstable -- once I had a couple boxes kernel panic in testing, I jumped off that boat immediately.
Moosefs is pragmatic -- written in pretty tight C, performant, and the web UI for seeing the state of the cluster and how files are replicating is very nice. We have a small cluster with maybe a dozen nodes running for almost 2 years now with no hiccups...
"Important CephFS currently lacks a robust ‘fsck’ check and repair function. Please use caution when storing important data as the disaster recovery tools are still under development. For more information about using CephFS today, see CephFS for early adopters"
RedHat is focusing on making the block and object storage bulletproof. They're content to let the opensource side twiddle away on ceph FS.
We do occasionally have issues where ALL storage nodes must be rebooted but other than it works great.
pretty hard to install (ceph-deploy makes it easier) but I've seen some pretty spiffy claims like really easy scaling and provisioning.
So wtf do we do when we need a 5TB shared storage platform across some EC2 nodes?
Shame though...