abronan

It’s a fair concern, and Mantissa is really just moving the complexity around.

One detail is that it relies on Strong Eventual Consistency, which is a slightly stronger model than Eventual Consistency. Replicas that have observed the same updates are guaranteed to converge to the same end state, regardless of the order in which those updates have been received.

That said, convergence alone doesn’t guarantee scheduling correctness. This is why it separates convergence from scheduling coordination. The CRDTs and MSTs provide the replicated state, and the scheduling protocol handles placement decisions and conflicting operations (with atomic slot reservation that is node local). There is a reconciliation loop (owned by a single node that is picked for the service/workload to deploy) that is then responsible for bringing the cluster back to its desired state if it potentially made a wrong call.

And it comes with trade-offs as mentioned in my post. You accept temporary divergences rather than requiring global agreement before every operation. This may or may not be acceptable for certain workloads. This removes the need for a quorum-based and centralized control plane, but it also means that certain invariants and guarantees have to be enforced at the scheduling and workload layer rather than by the datastore itself.

Strong consistency is also not a silver bullet to handle distributed systems. It is relatively hard to get it right and not serve stale values because of the presence of bugs in the protocol or state corruption. It could also bring the availability down and prevent from scheduling any more tasks due to a loss of quorum (which is by design and acceptable in most cases).

Mantissa is just making a different trade-off and accepting a different set of problems to solve: favoring availability and accepting that the desired state could shift temporarily on partition and network blips. If your cluster has good connectivity, then it should be a rare event, for which you should have prepared with some replicas that could handle being moved around. Again, it is not for everybody, and will probably remain a niche tool for users who want a simplified cluster deployment/updates to schedule mostly stateless tasks (even though stateful tasks are supported).

One remaining comment: replicated volumes do use a Raft quorum per volume. There are things that couldn't accept anything weaker than strong consistency, and volumes are one of them. So we are using strong eventual consistency for most of the state replication, but fallback to Raft for things that truly require strong consistency.

Hello HN,

I’m Alexandre, I have worked on Docker Swarm and the integration with distributed key value stores (etcd/consul/zookeeper), but also on Docker Swarmkit and the Raft distributed datastore, with the integration of etcd's Raft library.

Last July, I open sourced Mantissa (https://mantissa.io): a fully distributed scheduler and container orchestrator. I did this quietly to keep working on patches and stability. It is still at an experimental stage and will probably stay that way for a while, but could be tested to get a "look and feel" of the project.

Mantissa is using CRDTs[1] and Merkle Search Trees[2] for its data replication strategy (via anti-entropy/gossip). It is a bit of a departure from existing systems that are relying on strong consistency/linearizability (like Kubernetes, Swarm and Nomad): Mantissa focuses on fast convergence and self-stabilization instead. This comes with plenty of benefits (no quorum and Primary/Follower setup, simplified deployment and upgrades, etc.) but also trade-offs (increased storage size, no resource overcommit, temporary divergences in or outside of partition events, etc.) that would be too long to list and detail here.

Mantissa is a particular fit for small clusters, home labs, agent orchestration but could also be used in larger settings (I’d be comfortable to say up to a hundred nodes per cluster view/shard, but I’m making no claims for higher than this, since I don’t yet have access to hardware that would allow me to test and make such a claim). At the core was the idea that it should be trivial to interconnect machines together and create a transparent overlay on top of the operating system, regardless of the scale.

It is sort of a spiritual successor to Docker Swarmkit: going even further with simplification and reducing the deployment and maintenance burden. You'll see that it retains some of its design elements (with the node join flow) while taking inspiration from other systems like Google Omega[3].

Why building Mantissa while Kubernetes and other systems exist? I think it is worthwhile to experiment with new approaches and see how it may impact existing systems. Improving on the UX/DX of schedulers and orchestrators has been a long obsession of mine and Mantissa is a long story, with its initial codebase and architecture being 10 years old. The scope is incredibly large however and working on this alone is quite difficult even with the help of LLMs. AI is a double-edged sword, and I could write a full article around my frustrations with building complex distributed systems with LLMs (they have a hard time “reasoning” correctly about systems relying on convergence for example).

I’m having a lot of fun building this however and I hope this could be useful to some of you, despite its current limitations and experimental status. Feedback is also welcome to help steer the project in a direction where it could be the most useful/impactful to you.

Cheers.

Github: https://github.com/MantissaLabs/mantissa

[1] https://www.lip6.fr/Marc.Shapiro/papers/RR-7687.pdf [2] https://inria.hal.science/hal-02303490/file/paper%20%281%29.... [3] https://research.google/pubs/omega-flexible-scalable-schedul...

> but swarm was never intended to be a kubernetes contender.

Your comment is accurate for the original Swarm project, but a bit misleading regarding Swarm mode (released later on and integrated into docker).

I have worked on the original Swarm project and Swarmkit (on the distributed store/raft backend), and the latter was intended to compete with Kubernetes.

It was certainly an ambitious and borderline delusional strategy (considering the competition), but the goal was to offer a streamlined and integrated experience so that users wouldn't move away from Docker and use Swarm mode instead of Kubernetes (with a simple API, secured by default, just docker to install, no etcd or external key value metadata store required).

You can only go so far with a team of 10 people versus the hundreds scattered across Google/RedHat/IBM/Amazon, etc. There were so many evangelists and tech influencers/speakers rooting for Kubernetes already, reversing that trend was extremely difficult, even after initiating sort of a revolution in how developers deployed their apps with docker. The narrative that cluster orchestration was Google's territory (since they designed Borg that was used at a massive scale) was too entrenched to be challenged.

Swarm failed for many reasons (it was released too soon with a buggy experience and at an incomplete state, lacking a lot of the features k8s had, but also too late in terms of timing with k8s adoption). However, the goal for "Docker Swarm mode" was to compete with Kubernetes.

Oh no.. it has 3D Pinball Space Cadet and Doom, here goes my night :)

Amazing work on this website! It encourages exploration and navigating the folders to see all the content.

Since the article states that they're using Raft and not ZAB for the consensus algorithm and leader election, it must be less prone to bugs when it comes to electing a leader. Since Raft is easier to reason about and the leader election process is more straightforward (Raft minimizes the chance that any two nodes will be candidates at the same time and thus avoids starting multiple concurrent elections).

Thanks for this excellent article! Enjoyed it from start to finish. This gave me a good memory of the work we've done at docker embedding our own replicated and consistent metadata storage using etcd's raft library.

Looking at the initial pull request, is it correct that ClickHouse Keeper is based on Ebay's NuRaft library? Or did the Clickhouse team fork and modified this library to accommodate for ClickHouse usage and performance needs?

I wrote the initial implementation of the raft subsystem and it was definitely not a copy/paste. We started from scratch (using etcd's core raft) with the transport layer being grpc. My initial experiment could be found in this repository [1]. I then took the code from my initial experiment and included this into Swarmkit [2]. From there we went through many iterations on the initial code base and improved the UI with Docker swarm `init`/`join`/`leave` to make the experience of managing the cluster "friendly".

We spent quite some time evaluating different raft and paxos implementations (mostly Consul and etcd raft libraries), and found out etcd to be the most stable and flexible for our use case. It was very easy for example to swap the transport layer to use grpc. The fact that etcd implementation is represented as a simple state machine makes it also much easier to reason about under complex scenarios for debugging purposes, instead of digging into multiple layers of abstractions.

In retrospect, this came with quite a learning curve. We've had to deal with issues caused by our own misunderstandings on how to use the library properly. At the same time the fact that the developers favored stability as opposed to user friendliness was exactly what we found attractive using etcd's raft. Additionally, CoreOS developers were super friendly and helpful to help us fix these issues. We've reported and fixed some bugs as well. Kudos to them for all the help they provided at the time.

[1] https://github.com/abronan/proton [2] https://github.com/docker/swarmkit/commit/89de50f2092dfd2170...

This is without considering students whose parents have low revenue, thus getting access to financial help from the government. This was my case. Technically, I was paid to go to University, with 3600€ per year for being one of the top students.

I have been using Ghost for more than four years now (I tried using Wordpress before but I was left frustrated). Although I know there are probably attractive alternatives (Hugo/Netlify), I don't feel the urge to try something else. I like the Ghost editor and it provides with a nice and focused experience of writing [1].

I use a very simple, minimalist (albeit, slightly modified) theme called Oscar [2].

[1] https://abronan.com/ [2] https://github.com/abronan/oscar-ghost

In my French University, group or individual assignments had a very low coefficient compared to written exams (where it was almost impossible to cheat as this was basically like white board coding and students would be spaced by 2/3 seats between them).

It was in the order of 20%(group/coding)/80%(written exam). This had the effect of eliminating 50% of all students on the first semester for the first year of CompSci (DistSys) Master's Degree. So even if a group cheated on group/coding assignments, they most likely wouldn't go through the written exam. We called it: "The Purge".

From then, very few instances of cheating (topics were changed every year). One group got caught cheating on another one during the second year (iOS objectiveC class) and they got a failing grade.

Almost every professor would use automated code checkers on group and individual assignments. And if you get caught, you have to go through thorough questioning by the professor, eventually getting a failing grade and having to go through the second session of written exams (which was most of the time harder than the first session).

What was unavoidable though were group projects with only one person working on the project and the others doing nothing or just spending time preparing for written exams. This happened to me quite a few times. I didn't mind as I was learning a lot, but it got me exhausted for written exams where I was getting good but not outstanding grades.

Just talking about glassdoor, and their rankings are to take with a big grain of salt, especially for early stage startups and medium sized companies.

I worked for a company that had some issues with management and saw its glassdoor ranking plummet after several low scores were given by past employees with mention of harassment. They didn't claim the profile on glassdoor at the time. Unsurprisingly, after a few really negative reviews, they most likely encouraged current employees to post positive reviews to counter-balance for the purpose of claiming the profile. It was obvious because positive reviews were for most of them on the same date or very close to each other and came in bulk while reviews were spaced by weeks or months before (most of them rather negative).

The gist of it is that while negative reviews may not reflect the current state and culture of a company (disgruntled employees happen), positive reviews could also be misleading. And I'm not even accounting for people that have an interest for putting up positive reviews regardless of the working environment because they have big stakes into the company.

Best thing to do is to chat with several employees in different departments and ask for their honest opinion and share their experience. Generally people are open about the pros and cons of their own working environment if you talk to them behind closed doors.

This applies to the entire industry, we are often inflating our products with terms that do not describe the reality. Technical accuracy is important because it can drive a purchase decision when comparing features with a competitor. Companies and individuals invest time and money on these software/services, misleading them with inaccuracies can harm them directly or their business.

At least it's an honest statement from SpiderOak, it's better to fix a misuse of a term and admit an error than throwing a misleading term describing a product used by thousands of people and then delete it as if nothing happened.

When I see this post, I cannot help but think of how Docker described "Swarm mode" orchestration features during DockerCon 2016 using the terms "self-healing" and "self-organizing". Obviously, "Swarm mode" was neither "self-healing" nor "self-organizing" and a possibility is that they had no idea what those terms meant, but it looked good on paper and from a marketing point of view. While they have fixed it in the documentation after pointing this out internally, these terms have leaked in many blog posts and are still in plenty of talks recording on Youtube. It became hopeless to stop the spread of misinformation.

Despite this change, a lot of SpiderOak customers are still going to use the term Zero-Knowledge to describe the software to their friend/co-workers or business partners. The term will stick to them for awhile.

For having worked with Jessie while she was still working at Docker (I no longer work there either), I can confirm that. She is genuinely radiant and her excitement for technology in general is contagious.

She was always around to help or hack something cool. I remember the night of the release of Docker 1.9, we stayed late at the office to merge the last PRs and run test suites before releasing the binaries. Jess did not have to be there but she stayed to help Tibor (another amazing human being and prolific contributor) and myself 'til 23pm on the release process. You know you are surrounded with talented engineers when you can count on them regardless of the difficulties encountered. That night, her presence lifted a big burden off my shoulders.

I'm not sure I would have had the same excitement for Docker before joining if she was not working there. To me, she had a huge impact on the craze towards containers.

You should probably disclose that you work for Docker.

Looking at your profile/comments you seem to be openly hostile towards Kubernetes. On that note: this is quite representative of the mentality I encountered while working there. Open Source is hard and bashing other projects is not something I consider to be fair game (regardless of whether the other side is adopting that stance or not).

Meanwhile, I admire Hashicorp's attitude of focusing primarily on improving their product without taking part in this silly "orchestration war".

(and thanks to cpuguy83 for having a more sensible view of what it's like to work on Open Source projects)

Adding a quick opinion here: I worked for one of these companies (working with libraries written by all of these) and it's just a matter of personal taste. Docker, CoreOS, Hashicorp, etc. are "successful" (which is hard to confirm for now) because they can gather a lot of open source contributions thanks to the easy Go learning curve. I don't think this would be the case if they used Rust instead. Go is definitely not a play toy and you can build scalable and reliable software with it. The problem with Go in my opinion is also its ease of use, it's incredibly easy to write sloppy/buggy code with panics/data races that become increasingly hard to debug. I've had my fair share of frustrating experiences debugging concurrent code with hundreds of goroutines or investigating random data races on complex distributed systems code. I've yet to see that happening with Rust after four months of using it, the compiler does an incredible job at preventing these.

While Go's frustration often comes from debugging sloppy concurrent code, the frustration with Rust is often about having your code to compile. But once it compiles, it runs beautifully. Some prefer the former, I personally prefer the latter. Also, I think the argument of increased productivity with Go should not be the main argument, because the time you save writing a large program, you will probably lose trying to debug a deadlock (which could still happen in Rust but harder to make happen) or a random data race that the race detector couldn't catch (or in a vendored library).

If I had to define the experience with both languages, Go is the instant gratification option, you feel productive quickly, the language is easy to learn with a forgiving syntax sometimes at the expense of correctness. Rust is the delayed gratification option, frustrating at times, very hard to learn and master but offering strong safety guarantees with an elegant syntax (which is a personal opinion).

Also it's a controversial topic in the Go community but the lack of generics...

Rust has its downsides too, for example the slow compile times or the lack of maintained libraries for some core functionalities but overall, programming with Rust is a much more enjoyable experience once you start getting ahead the learning curve.

One last thing: It is hopeless trying to learn Rust in four days without very good resources/books. I recommend "Programming Rust" from OReilly if you really want to dig into the language. It's incomplete at the time of writing but still one of the most useful resource out there (with the free Rust ebook). My knowledge of Rust dramatically increased after reading it and I felt much more confident writing larger programs.

I definitely don't get stressed when people are pointing out errors in my code. It's a sign that you can trust your coworkers. As some people mentioned already, I become stressed when people just approve a big and complex change without any question/comment because this is likely that they are busy and that they didn't try to understand it. This is how you get bugs slipping into production environment.

Unnecessary nitpicks that are holding Pull Requests for days/weeks are stressful though. Because they are holding a feature hostage to satisfy the ego of the commenter on an opinion that is subjective. It is especially unnerving when the commenter is dropping the nitpick and then disappears and never comments on the PR again, or only after months. I appreciate when the commenter clearly points out that this could be an optional change to slightly improve the code (or by opening a separate issue after merging).

I maintain a list of my readings triaged by topics: https://github.com/abronan/readings

This year was rough for me as I had to deal with severe symptoms of anxiety that eventually led to panic attacks. I tried to understand the phenomenon and tackle subjects such as anxiety, consciousness and perceptual experience. Three books are especially interesting in that regard:

- The Feeling of What Happens: Body and Emotion in the Making of Consciousness - Antonio Damasio - https://www.goodreads.com/book/show/125777.The_Feeling_of_Wh...

- Anxious: Using the Brain to Understand and Treat Fear and Anxiety - Joseph LeDoux - https://www.goodreads.com/book/show/23398722-anxious?ac=1&fr...

- Seeing Things As They Are: A Theory of Perception - John Searle: https://www.goodreads.com/book/show/22104265-seeing-things-a...

The first two books are dealing with the neurophysiological aspects, with a focus on the brain and the specific areas involved with physical sensory experiences and/or consciousness. It explains how some areas of the brain are linked to fear and anxiety (fear conditioning, fight-or-flight response, etc.). Reading "The Feeling of What Happens" gives you all the necessary knowledge to fully understand the second book which is a tough read. The book "Anxious" also gives you a glimpse on different methods to treat and prevent symptoms of anxiety (Cognitive behavioral Therapy, SSRI, beta blockers, meditation, etc..). The last book is theoretical but comes as a good complement and gives you a broad understanding on the notion of perception (which is central to the first book).

Knowing which parts of the brain are involved with fear and anxiety and how everything fits together helps me controlling my emotions when physical symptoms of anxiety are appearing (the trigger to panic attacks).

I stumbled upon this tutorial: https://software.intel.com/sites/default/files/managed/54/8b... it seems that you can at least assign co-processors to KVM virtual machines.