News for hackers. Stuff that matters.
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @12:10PM (#28258071)
It uses Raft underneath as well which means there's a bunch of non-determinism and hell for anyone who invokes it as well from experience. The thing is cursed.

Source: several years dealing with vault and consul.

Re:IceFireDB: Distributed disk storage database based on Raf... by tempest_ on Saturday August 21, 2021 @12:40PM (#28258360)
We use consul a bit as some "light" service discovery and a KV store for a few things without much issue so far.

What demons did you encounter with vault/consul?

Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @12:44PM (#28258395)
We had some massive problems including complete cluster collapse requiring rebuilds from scratch, eternal leadership elections and occasionally nodes would just entirely stop responding to KV requests causing cascading failures outside. Vault is a massive damage multiplier for these issues plus some other nasty ones like buggy barely supported plugins.
Re:IceFireDB: Distributed disk storage database based on Raf... by zetsurin on Saturday August 21, 2021 @07:05PM (#28261361)
Wouldn't mind knowing what versions you were using/how long ago this was. We haven't seen anything like this in > 3 years. Possible a barely supported plugin caused issues?
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @07:29PM (#28261529)
This is post 1.0 and without any plugins for consul. Vault has been a persistent point of failure.

It’s a good idea in principle but it’s not got a good ROI

Re:IceFireDB: Distributed disk storage database based on Raf... by chucky_z on Saturday August 21, 2021 @11:02PM (#28262502)
Huh. I've been running Consul since 0.7 or so, and the only time any of this happened was my own fault.

Most of the problems I've had with Vault have been around it's Terraform provider which they've improved enough that it's not an issue anymore.

I think the only thing about Raft that folks don't realize is how disk hungry it gets, if you want fast write performance you gotta make dang sure all those fsyncs can keep up. Our largest Consul cluster today runs on storage-heavy boxes as it does ~500Mb/s of writes pretty much 24/7.

I use Consul w/ Vault today instead of the internal storage for Vault just cause Consul has really nice monitoring around some stuff that Vault doesn't (path-based stuff for the most part), I think the internal storage is a really good option for 90% of use-cases.

Re:IceFireDB: Distributed disk storage database based on Raf... by cortesoft on Saturday August 21, 2021 @01:48PM (#28258960)
What is a better consensus protocol to use?
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @02:01PM (#28259097)
Look higher up the problem domain and solve it without requiring a consensus protocol.
Re:IceFireDB: Distributed disk storage database based on Raf... by rubiquity on Saturday August 21, 2021 @08:21PM (#28261830)
Not sure why you’re being down voted as this is definitely the path forward. The industry and research have both explored the polar opposites of weak consistency protocols and linearizibility and consensus on the other end. Understanding your domain and knowing how you can step down to something weaker than consensus will be critical for future applications.
Re:IceFireDB: Distributed disk storage database based on Raf... by cortesoft on Saturday August 21, 2021 @03:08PM (#28259629)
Hmm, how do you have high availability consistent data without a consensus protocol? No matter where in the problem chain you move, you have to eventually solve that problem.
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @07:25PM (#28261496)
Easy you push the configuration to every target. Then they are logically consistent.

We did this for 30 years fine before someone invented this stack on deployments much larger then the average consul or vault deployment these days.

I had something running 15,000 dynamic rps on Apache about 15 years ago.

People are blinded from simplicity by complexity. Eventually complexity owns you. You can only own simplicity.

At the end of the day this is one way to solve a problem that doesn’t need to be solved that someone has convinced you is a problem.

Re:IceFireDB: Distributed disk storage database based on Raf... by cortesoft on Saturday August 21, 2021 @08:00PM (#28261713)
People need distributed consistent data for a lot of things besides configuration.
Re:IceFireDB: Distributed disk storage database based on Raf... by yencabulator on Monday August 23, 2021 @12:28PM (#28278337)
People say they do, but then they proceed to read stale data from replicas all the time...
Re:IceFireDB: Distributed disk storage database based on Raf... by asadawadia on Saturday August 21, 2021 @07:39PM (#28261598)
>Easy you push the configuration to every target. Then they are logically consistent.

What does this mean? This doesn't mean anything?

Are you saying to push the DB to the client?

Re:IceFireDB: Distributed disk storage database based on Raf... by caffeine on Sunday August 22, 2021 @07:06PM (#28270593)
I think he’s saying you have a cronjob with an rsync to every client
Re:IceFireDB: Distributed disk storage database based on Raf... by PYTHONDJANGO on Saturday August 21, 2021 @12:44PM (#28258399)
Please post URLs to bugs / issues to give your comment some cred. Thanks!