News for hackers. Stuff that matters.
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @12:44PM (#28258395)
We had some massive problems including complete cluster collapse requiring rebuilds from scratch, eternal leadership elections and occasionally nodes would just entirely stop responding to KV requests causing cascading failures outside. Vault is a massive damage multiplier for these issues plus some other nasty ones like buggy barely supported plugins.
Re:IceFireDB: Distributed disk storage database based on Raf... by zetsurin on Saturday August 21, 2021 @07:05PM (#28261361)
Wouldn't mind knowing what versions you were using/how long ago this was. We haven't seen anything like this in > 3 years. Possible a barely supported plugin caused issues?
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @07:29PM (#28261529)
This is post 1.0 and without any plugins for consul. Vault has been a persistent point of failure.

It’s a good idea in principle but it’s not got a good ROI

Re:IceFireDB: Distributed disk storage database based on Raf... by chucky_z on Saturday August 21, 2021 @11:02PM (#28262502)
Huh. I've been running Consul since 0.7 or so, and the only time any of this happened was my own fault.

Most of the problems I've had with Vault have been around it's Terraform provider which they've improved enough that it's not an issue anymore.

I think the only thing about Raft that folks don't realize is how disk hungry it gets, if you want fast write performance you gotta make dang sure all those fsyncs can keep up. Our largest Consul cluster today runs on storage-heavy boxes as it does ~500Mb/s of writes pretty much 24/7.

I use Consul w/ Vault today instead of the internal storage for Vault just cause Consul has really nice monitoring around some stuff that Vault doesn't (path-based stuff for the most part), I think the internal storage is a really good option for 90% of use-cases.