News for hackers. Stuff that matters.
Re:Intel Shuts Down Lustre File System Business by pinewurst on Friday April 21, 2017 @11:51AM (#14166622)
It's been amusing to see all the apologia and simple ass-covering following this.

Lustre is a monstrosity - badly designed, poorly implemented, very hard to configure and keep running or even get adequate performance under other than a single limited use case.

Good riddance to bad rubbish.

Re:Intel Shuts Down Lustre File System Business by arcanus on Friday April 21, 2017 @12:21PM (#14166926)
> or even get adequate performance under other than a single limited use case.

Hmm?

From the article: " In the core HPC market, Lustre has 75 of the Top 100 systems on the Top 500 rankings, and is used on nine of the top ten systems."

That is a strong argument for it being performant. That 'single limited use case' you mention is... high performance computing.

Re:Intel Shuts Down Lustre File System Business by pinewurst on Friday April 21, 2017 @01:55PM (#14167714)
That's an artifact of the limited choices available. Plus the core HPC market is both notoriously inbred and tolerant of low system reliability and high delicacy that the enterprise wouldn't let in the door.

And no, "single limited use case" I mean, is a relatively small number of very large files that need to be sequentially streamed out/in as fast as possible. That's a small subset of HPC.

GPFS is the gold standard but it's expensive.

Lustre is more a byproduct of the HPC vendors trying to get more margin by using open source. They've drunk from the poisoned chalice in that the amount of time, money and effort required to get Lustre to acceptability (limited as that may be) has been far more than GPFS license/support cost. Even from the brain-eating zombie that is today's IBM.

Re:Intel Shuts Down Lustre File System Business by lo_stronzo on Friday April 21, 2017 @03:24PM (#14168566)
> GPFS is the gold standard but it's expensive.

Confirming! And going on a small tangent (ha!).

Our previous configuration was Lustre and XFS/NFS; the former was the scratch file system for HPC applications and the latter for home directories and what not.

Lustre was definitely a beast (in a good sense), but we'd occasionally get bit by a work load high in metadata operations - which would bring Lustre to its knees due to latency.

XFS/NFS was great for its purpose (no HPC workloads), but we'd also get bit by a user or users reading seemingly tiny, innocuous files (inadvertently in parallel) which would cause load averages to spike; surprisingly the latency wasn't as bad as the Lustre workload mentioned above.

Not too drink the GPFS kool aid here, but it's definitely solved both problems above. It has its issues, but definitely handles the common I/O patterns seen on our cluster.

Re:Intel Shuts Down Lustre File System Business by pinewurst on Friday April 21, 2017 @04:42PM (#14169275)
If I walked you through the Lustre metadata state machine you wouldn't be surprised by Lustre latency any longer. It's a veritable Rube Goldberg machine without the amusement.
Re:Intel Shuts Down Lustre File System Business by lo_stronzo on Friday April 21, 2017 @06:37PM (#14169971)
I believe it.

At least the Lustre developers (pre-Intel) had the foresight to enable extremely good debugging - you could simply enable a few procfs settings and easily find the offenders.

It's still amusing to me however, that the biggest offenders of Lustre slow-downs were single core processes. I'd check the RPC counts per node, find the violator, and then check per PID statistics; it was always (95%+) a single core application performing thousands of calls.

We never did experiment with the 2.x branch of the software. I recall one of our co-workers stating that even the developers did not believe the dual MDS set-up was production ready at that time.

Re:Intel Shuts Down Lustre File System Business by convolvatron on Friday April 21, 2017 @12:51PM (#14167207)
right, so they took a vertical that no one else wanted and served it kind of badly. I don't know about the last few years, but Lustre used to be famous for eating up a huge amount of time in the field, even given that its only supposed to be relevant for really large files.

it definitely served a need that nothing else did, but from an armchair position we could collectively do better

Re:Intel Shuts Down Lustre File System Business by didip on Friday April 21, 2017 @12:05PM (#14166759)
In your opinion, what would be better than Lustre? Ceph, Gluster, HDFS?
Re:Intel Shuts Down Lustre File System Business by koolba on Friday April 21, 2017 @12:27PM (#14166987)
Those choices are apples and oranges. Particularly HDFS as you can't really use it as a native fs.
Re:Intel Shuts Down Lustre File System Business by khm on Friday April 21, 2017 @12:39PM (#14167106)
GPFS.
Re:Intel Shuts Down Lustre File System Business by yourapostasy on Friday April 21, 2017 @11:52AM (#14166635)
Any recommendations for alternatives, and what their relative weak points are in your experience?
Re:Intel Shuts Down Lustre File System Business by zhynn on Friday April 21, 2017 @02:36PM (#14168100)
I only have experience running Lustre and Gluster. Gluster was a mess, to be honest, though that was several years ago. Lustre (and we run Intel Enterprise Lustre) has been pretty solid. Most HPC outfits run lustre over ZFS, actually, so you get the benefits of both.

Ceph and Gluster can't achieve the kind of performance required for HPC scratch. Hadoop can, but if you are managing a general purpose HPC cluster, nearly all software out there is expecting to write to a posix filesystem. Much of this software is built out of old C and FORTRAN that is not going to be rewritten. Researchers are not software engineers. You need a "lowest common denominator" of storage that you can read/write at high speed in parallel. There are few technologies for this particular use case.

The others that I know of (but haven't tried) are:

- OrangeFS (http://www.orangefs.org/) - BeeGFS (http://www.beegfs.com/) - OpenAFS/Arla (ancient, and weird, it is one of the big reasons Scientific Linux exists as a distro: http://www.openafs.org/, http://www.stacken.kth.se/project/arla/) - Scality (http://www.scality.com/) - GPFS (Spectrum now: https://www-03.ibm.com/systems/storage/spectrum/scale/) - Qumulo (they say they can hit the numbers, but I have doubts: http://qumulo.com/

There are probably many more, but these are the ones I know about.

Re:Intel Shuts Down Lustre File System Business by __jal on Friday April 21, 2017 @06:00PM (#14169763)
People may use Gluster for HPC; I don't know. But it really isn't suited.

I run two Gluster clusters for more boring enterprisey reliability.

It is still kind of a mess - there are rough edges all over. Log messages make zero sense by themselves - you either need to learn what they're indicative of or read the code. Frequently, typoing configuration commands (configuration is mostly imperative command line driven, not via configuration files) so that it fails, actually does make changes, such that you'll have to unwind what it did before you can try again, etc.

All that said, Gluster is pretty solid for our sue - distributed, fault-tolerant POSIX storage. But I wouldn't use it in an HPC environment.

Re:Intel Shuts Down Lustre File System Business by pinewurst on Friday April 21, 2017 @03:05PM (#14168376)
Qumulo can't hit the numbers. It's just NFS, running on ho-hum hardware with an Ethernet cluster interconnect. It's like slow Isilon if cheaper and with less functionality.
Re:Intel Shuts Down Lustre File System Business by takeda on Friday April 21, 2017 @12:44PM (#14167152)
So far I heard a lot about GlusterFS and Ceph being great but so far I only seen MooseFS in production.

MooseFS seems to do its job well, the only problem is that by default it lacks HA. The master server is very important since it keeps track of where all data is located, so if it is gone your data is gone. You can set up one or more metalogger nodes, which sole purpose is to backup the metadata information. You can also manually promote metalogger to master.

They have commercial offering that includes HA, but I never used so hard to comment about it.

You could also set up HA yourself using corosync and peacemaker, but it's a bit challenging task.

Re:Intel Shuts Down Lustre File System Business by zhynn on Friday April 21, 2017 @02:36PM (#14168104)
Oh yeah, MooseFS. I forgot about that one.
Re:Intel Shuts Down Lustre File System Business by photon-torpedo on Friday April 21, 2017 @12:17PM (#14166884)
Any opinion on BeeGFS?
Re:Intel Shuts Down Lustre File System Business by usernam on Friday April 21, 2017 @12:47PM (#14167165)
We used BeeGFS since it was called FhGFS on a cluster of 32 nodes for a total capacity ranging from 256tb-1Pb. BeeGFS is very easy to configure and maintain, outperforming Lustre for every task we threw it at. In fact, we never transitioned lustre into production. Lustre was often slower than plain NFS for the cases where all storage could be mounted by a single node(!).

Unfortunately, it's prior licensing scheme prevented larger initial adoption. A good chunk of large scale setups I've seen have adopted Ceph, but I cannot comment on performance.

Re:Intel Shuts Down Lustre File System Business by pinewurst on Friday April 21, 2017 @01:50PM (#14167677)
BeeGFS is great. I originally evaluated it when it was FhGFS and am still involved with a large production BeeGFS cluster deployment.

Any knucklehead can install it and it works transparently with excellent performance. It's currently not enterprise-grade fault tolerant - that's the worst thing I can say about it.

Re:Intel Shuts Down Lustre File System Business by hpcjoe on Friday April 21, 2017 @02:58PM (#14168322)
> It's currently not enterprise-grade fault tolerant - that's the worst thing I can say about it.

Its getting there quickly. BeeGFS v6 has metadata mirroring via buddy groups, and data mirroring the same way. A few things are still not there, but it is rapidly getting closer.

Being open source helps as well.

Definitely recommend it, and have deployed it and supported it many places in my previous life: http://scalableinformatics.com .