News for hackers. Stuff that matters.
Re:IceFireDB: Distributed disk storage database based on Raf... by maxpert on Saturday August 21, 2021 @11:24AM (#28257724)
A database without any test harness? While this could be a good toy or PoC I would never use it in production. Readers should be aware, just because it's on HN doesn't mean it's production ready.
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @12:10PM (#28258071)
It uses Raft underneath as well which means there's a bunch of non-determinism and hell for anyone who invokes it as well from experience. The thing is cursed.

Source: several years dealing with vault and consul.

Re:IceFireDB: Distributed disk storage database based on Raf... by tempest_ on Saturday August 21, 2021 @12:40PM (#28258360)
We use consul a bit as some "light" service discovery and a KV store for a few things without much issue so far.

What demons did you encounter with vault/consul?

Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @12:44PM (#28258395)
We had some massive problems including complete cluster collapse requiring rebuilds from scratch, eternal leadership elections and occasionally nodes would just entirely stop responding to KV requests causing cascading failures outside. Vault is a massive damage multiplier for these issues plus some other nasty ones like buggy barely supported plugins.
Re:IceFireDB: Distributed disk storage database based on Raf... by zetsurin on Saturday August 21, 2021 @07:05PM (#28261361)
Wouldn't mind knowing what versions you were using/how long ago this was. We haven't seen anything like this in > 3 years. Possible a barely supported plugin caused issues?
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @07:29PM (#28261529)
This is post 1.0 and without any plugins for consul. Vault has been a persistent point of failure.

It’s a good idea in principle but it’s not got a good ROI

Re:IceFireDB: Distributed disk storage database based on Raf... by chucky_z on Saturday August 21, 2021 @11:02PM (#28262502)
Huh. I've been running Consul since 0.7 or so, and the only time any of this happened was my own fault.

Most of the problems I've had with Vault have been around it's Terraform provider which they've improved enough that it's not an issue anymore.

I think the only thing about Raft that folks don't realize is how disk hungry it gets, if you want fast write performance you gotta make dang sure all those fsyncs can keep up. Our largest Consul cluster today runs on storage-heavy boxes as it does ~500Mb/s of writes pretty much 24/7.

I use Consul w/ Vault today instead of the internal storage for Vault just cause Consul has really nice monitoring around some stuff that Vault doesn't (path-based stuff for the most part), I think the internal storage is a really good option for 90% of use-cases.

Re:IceFireDB: Distributed disk storage database based on Raf... by cortesoft on Saturday August 21, 2021 @01:48PM (#28258960)
What is a better consensus protocol to use?
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @02:01PM (#28259097)
Look higher up the problem domain and solve it without requiring a consensus protocol.
Re:IceFireDB: Distributed disk storage database based on Raf... by rubiquity on Saturday August 21, 2021 @08:21PM (#28261830)
Not sure why you’re being down voted as this is definitely the path forward. The industry and research have both explored the polar opposites of weak consistency protocols and linearizibility and consensus on the other end. Understanding your domain and knowing how you can step down to something weaker than consensus will be critical for future applications.
Re:IceFireDB: Distributed disk storage database based on Raf... by cortesoft on Saturday August 21, 2021 @03:08PM (#28259629)
Hmm, how do you have high availability consistent data without a consensus protocol? No matter where in the problem chain you move, you have to eventually solve that problem.
Re:IceFireDB: Distributed disk storage database based on Raf... by hughrr on Saturday August 21, 2021 @07:25PM (#28261496)
Easy you push the configuration to every target. Then they are logically consistent.

We did this for 30 years fine before someone invented this stack on deployments much larger then the average consul or vault deployment these days.

I had something running 15,000 dynamic rps on Apache about 15 years ago.

People are blinded from simplicity by complexity. Eventually complexity owns you. You can only own simplicity.

At the end of the day this is one way to solve a problem that doesn’t need to be solved that someone has convinced you is a problem.

Re:IceFireDB: Distributed disk storage database based on Raf... by cortesoft on Saturday August 21, 2021 @08:00PM (#28261713)
People need distributed consistent data for a lot of things besides configuration.
Re:IceFireDB: Distributed disk storage database based on Raf... by yencabulator on Monday August 23, 2021 @12:28PM (#28278337)
People say they do, but then they proceed to read stale data from replicas all the time...
Re:IceFireDB: Distributed disk storage database based on Raf... by asadawadia on Saturday August 21, 2021 @07:39PM (#28261598)
>Easy you push the configuration to every target. Then they are logically consistent.

What does this mean? This doesn't mean anything?

Are you saying to push the DB to the client?

Re:IceFireDB: Distributed disk storage database based on Raf... by caffeine on Sunday August 22, 2021 @07:06PM (#28270593)
I think he’s saying you have a cronjob with an rsync to every client
Re:IceFireDB: Distributed disk storage database based on Raf... by PYTHONDJANGO on Saturday August 21, 2021 @12:44PM (#28258399)
Please post URLs to bugs / issues to give your comment some cred. Thanks!
Re:IceFireDB: Distributed disk storage database based on Raf... by corerman on Friday August 27, 2021 @04:04AM (#28325399)
Project tests have been added, from open source community partners, thanks for the initial attention.
Re:IceFireDB: Distributed disk storage database based on Raf... by tomnipotent on Saturday August 21, 2021 @01:12PM (#28258659)
A database without any code, actually.

It's less than a few hundred lines of Go that just wraps two other databases (syndtr/goleveldb and ledisdb/ledisdb) with a third library (tidwall/uhaha) that provides a Raft API.

Re:IceFireDB: Distributed disk storage database based on Raf... by 3np on Saturday August 21, 2021 @02:08PM (#28259147)
Oooh that means I can form it to do Redis instead right? Because that could be a nice way to to Redis clustering
Re:IceFireDB: Distributed disk storage database based on Raf... by tomnipotent on Saturday August 21, 2021 @02:58PM (#28259545)
I suppose you could stick tidwall/uhaha directly in front of redis, but I'm not entirely certain what you'd call that...

Here's the LSET code:

https://github.com/gitsrc/IceFireDB/blob/main/lists.go#L232

    func cmdLSET(m uhaha.Machine, args []string) (interface{}, error) {
     if len(args) != 4 {
      return nil, rafthub.ErrWrongNumArgs
     }
    
     index, err := ledis.StrInt64([]byte(args[2]), nil)
     if err != nil {
      return nil, err
     }

     if err := ldb.LSet([]byte(args[1]), int32(index), []byte(args[3])); err != nil {
      return nil, err
     }
     return redcon.SimpleString("OK"), nil
    }
So what "IceFireDB" is:

1. tidwall/uhaha - Raft server (m uhaha.Machine, rafthub)

2. tidwall/redcon - Read/write redis protocol (redcon.SimpleString)

3. ledisdb/ledisdb - Redis-compatible with disk persistence via leveldb (ldb.LSet)

4. syndtr/goleveldb/leveldb - Provides snapshots, other scattered references throughout code

It also includes this seemingly random file below, which seems to implement some string slice overloads using unsafe.Pointer:

https://github.com/siddontang/go/blob/master/hack/hack.go

    // no copy to change slice to string
    // use your own risk
    func String(b []byte) (s string) {
     pbytes := (*reflect.SliceHeader)(unsafe.Pointer(&b))
     pstring := (*reflect.StringHeader)(unsafe.Pointer(&s))
     pstring.Data = pbytes.Data
     pstring.Len = pbytes.Len
     return
    }

    // no copy to change string to slice
    // use your own risk
    func Slice(s string) (b []byte) {
     pbytes := (*reflect.SliceHeader)(unsafe.Pointer(&b))
     pstring := (*reflect.StringHeader)(unsafe.Pointer(&s))
     pbytes.Data = pstring.Data
     pbytes.Len = pstring.Len
     pbytes.Cap = pstring.Len
     return
    }
Re:IceFireDB: Distributed disk storage database based on Raf... by sitkack on Sunday August 22, 2021 @09:58AM (#28265503)
Do you need Redis or the Redis protocol?

Aerospike and ScyllaDB both support a subset of the Redis API and run as a durable cluster. Both have been tested with Jepsen.

Re:IceFireDB: Distributed disk storage database based on Raf... by tomnipotent on Sunday August 22, 2021 @04:38PM (#28269370)
> Do you need Redis or the Redis protocol?

Only if you want to be a drop-in replacement and take advantage of existing compatible libraries. With the recent tea around the official Elasticsearch Python library, it becomes a more interesting question.