charleshn

Thanks for the article.

I have two questions/comments:

1. The N-ring fade-in is quite neat. I guess without the constraint of hash parity rendezvous hashing [0] could have been an elegant approach since it has support for weights (and generally better statistical properties than consistent hashing based on rings).

2. You mention still having the fallback of your existing load balancer. Is this a temporary thing during rollout or do you intend to keep it long-term. Asking because I generally tend to stay clear of fallbacks in distributed systems, as they introduce bimodality and metastable failures [1] [2].

[0] https://en.wikipedia.org/wiki/Rendezvous_hashing

[1] https://builder.aws.com/content/3EuS9Sakq7L3VLQIF3qzfMfke1Y/...

[2] https://brooker.co.za/blog/2021/05/24/metastable.html

> Some people like to parrot "next token prediction", "LLMs can only interpolate", and other nonsense

Thank you for illustrating my point.

Yes, they can.

Some people like to parrot "next token prediction", "LLMs can only interpolate", and other nonsense, but it is obviously not true for many reasons, in particular since we introduced RL.

Humans do not have the monopoly on generating novel ideas, modern AI models using post training, RL etc can come to them in the same way we do, exploration.

See also verifier's law [0]: "The ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI."

This applied to chess, go, strategy games, and we can now see it applying to mathematics, algorithmic problems, etc.

It is incredibly humbling to see AI outperform humans at creative cognitive tasks, and realise that the bitter lesson [1] applies so generally, but here we are.

[0] https://www.jasonwei.net/blog/asymmetry-of-verification-and-...

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Not sure what you mean: OpenEvolve is an open source implementation of AlphaEvolve: https://huggingface.co/blog/codelion/openevolve

They'll likely make it available at some point, but for now one can use OpenEvolve [0] which is not quite as good but should be a good start to use the same LLM-driven evolutionary framework.

[0] https://github.com/algorithmicsuperintelligence/openevolve

> Application pods learn the current assignment through a library called the Slicelet (S for server side). The Slicelet maintains a local cache of the latest assignment by fetching it from the Dicer service and watching for updates. When it receives an updated assignment, the Slicelet notifies the application via a listener API.

For a critical control plane component like this, I tend to prefer a constant work pattern [0], to avoid metastable failures [1], e.g. periodically pull the data instead of relying on notifications.

[0] https://aws.amazon.com/builders-library/reliability-and-cons...

[1] https://brooker.co.za/blog/2021/05/24/metastable.html

They should be reintroducing the 3D vcache [0] variants (X) in EPYC, with a higher cache/core ratio, that was present in EPYC4 (e.g. 9684X [1]) they for some reason wasn't available in EPYC5.

Makes a massive difference at high density and utilisation, with the standard cache/core performance can really degrade under load.

[0] https://www.amd.com/en/products/processors/technologies/3d-v...

[1] https://www.amd.com/en/products/processors/server/epyc/4th-g...

It's fundamentally because of verifier's law [0].

Current AI, and in particular RL-based, is already or will soon achieve super human performance on problems that can be - quickly - verified and measured.

So maths, algorithms, etc and well defined bugs fall into that category.

However architectural decision, design, long-term planning where there are few data points, no model and long iteration cycles are not so much amenable to it.

[0] https://www.jasonwei.net/blog/asymmetry-of-verification-and-...

> std::hardware_destructive_interference_size Exists so you don't have to guess, although in practice it'll basically always be 64.

Unfortunately it's not quite true, do to e.g. spacial prefetching [0]. See e.g. Folly's definition [1].

[0] https://community.intel.com/t5/Intel-Moderncode-for-Parallel...

[1] https://github.com/facebook/folly/blob/d2e6fe65dfd6b30a9d504...

> There's a good reason so much research is done on Nvidia clusters and not TPU clusters.

You are aware that Gemini was trained on TPU, and that most research at Deepmind is done on TPU?

I can relate.

I had tinnitus for over 10 years. My tinnitus was not the usual ringing type, it was some sort of humming, low frequency noise. The frequency was not constant, it could vary. It could sometimes stop for 5-10 minutes, e.g. after a hot bath.

Went to see many specialists, tried everything, to no avail.

One day I started experiencing recurrent tension and light pain in my neck and shoulder blades, so I started doing some neck and shoulder blades stretches several times a day.

After a few weeks, the pain was gone, and I realised the tinnitus had stopped. This was maybe 2 years ago (I am still doing those exercises multiple times a day).

A few questions if the authors are around!

> Is hardware agnostic and uses TCP/IP to communicate.

So no RDMA? It's very hard to make effective use of modern NVMe drives bandwidth over TCP/IP.

> A logical shard is further split into five physical instances, one leader and four followers, in a typical distributed consensus setup. The distributed consensus engine is provided by a purpose-built Raft-like implementation, which we call LogsDB

Raft-like, so not Raft, a custom algorithm? Implementing distributed consensus correctly from scratch is very hard - why not use some battle-tested implementations?

> Read/write access to the block service is provided using a simple TCP API currently implemented by a Go process. This process is hardware agnostic and uses the Go standard library to read and write blocks to a conventional local file system. We originally planned to rewrite the Go process in C++, and possibly write to block devices directly, but the idiomatic Go implementation has proven performant enough for our needs so far.

The document mentions it's designed to reach TB/s though. Which means that for an IO intensive workload, one would end up wasting a lot of drive bandwidth, and require a huge number of nodes.

Modern parallel filesystems can reach 80-90GB/s per node, using RDMA, DPDK etc.

> This is in contrast to protocols like NFS, whereby each connection is very stateful, holding resources such as open files, locks, and so on.

This is not true for NFSv3 and older, it tends to be stateless (no notion of open file).

No mention of the way this was developed and tested - does it use some formal methods, simulator, chaos engineering etc?

Interesting that neither the article nor the comments mention the CALM theorem [0], which gives a framework to explain when coordination-free consistency is possible, and is arguably the big idea behind SEC.

[0] https://arxiv.org/abs/1901.01930

You can have a look at the DeepSeek paper, in particular section "2.2 DeepSeek-R1-Zero: Reinforcement Learning on the Base Mode".

But generally the idea is that it's, you need some notion of reward, verifiers etc.

Works really well for maths, algorithms, amd many things actually.

See also this very short essay/introduction: https://www.jasonwei.net/blog/asymmetry-of-verification-and-...

That's why we have IMO gold level models now, and I'm pretty confident we'll have superhuman mathematics, algorithmic etc models before long.

Now domains which are very hard to verify - think e.g. theoretical physics etc - that's another story.

> We cannot add more compute to a given compute budget C without increasing data D to maintain the relationship. > We must either (1) discover new architectures with different scaling laws, and/or (2) compute new synthetic data that can contribute to learning (akin to dreams).

Of course we can, this is a non issue.

See e.g. AlphaZero [0] that's 8 years old at this point, and any modern RL training using synthetic data, e.g. DeepSeek-R1-Zero [1].

[0] https://en.m.wikipedia.org/wiki/AlphaZero

[1] https://arxiv.org/abs/2501.12948

Yes, 450GB/s is the per GPU bandwidth in the nvlink domain. 3.2Tbps is the per-host bandwidth in the scale out IB/Ethernet domain.

Could you check the value of your kernel's net.ipv4.tcp_slow_start_after_idle sysctl, and if it's non zero set it to 0?

You can now add getting gold at IMO [0] to the above list.

[0] https://x.com/alexwei_/status/1946477742855532918

Frontier models went from not being able to count the number of 'r's in "strawberry" to getting gold at IMO in under 2 years [0], and people keep repeating the same clichés such as "LLMs can't reason" or "they're just next token predictors".

At this point, I think it can only be explained by ignorance, bad faith, or fear of becoming irrelevant.

[0] https://x.com/alexwei_/status/1946477742855532918

I mentioned algorithms, not software engineering, precisely for that reason.

But the next step is obviously increased formalism via formal methods, deterministic simulators etc, basically so that one could define an environment for a RL agent.

I'm always surprised by the number of people posting here that are dismissive of AI and the obvious unstoppable progress.

Just looking at what happened with chess, go, strategy games, protein folding etc, it's obvious that pretty much any field/problem that can be formalised and cheaply verified - e.g. mathematics, algorithms etc - will be solved, and that it's only a matter of time before we have domain-specific ASI.

I strongly encourage everyone to read about the bitter lesson [0] and verifier's law [1].

[0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

[1] https://www.jasonwei.net/blog/asymmetry-of-verification-and-...

We do already have ASICs, see Google's TPU to get some cost estimates.

HBM is also very expensive.

You might want to look into TCP BBR [0], it might help. Easy to try on Linux, simple sysctl.

[0] https://en.m.wikipedia.org/wiki/TCP_congestion_control#TCP_B...

As can be seen from other comments, people tend to focus on the consistency implications, but something not discussed often in the context of distributed systems is that caches tend to introduce bimodality and metastability [0] [1]. See e.g. DynamoDB for an example of design taking it into account [2].

[0] https://brooker.co.za/blog/2021/08/27/caches.html

[1] https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s...

[2] https://brooker.co.za/blog/2022/07/12/dynamodb.html

See e.g. https://www.alibabacloud.com/blog/async-fork-mitigating-quer...

> Fork on Linux should use copy-on-write vmpages now, so if you fork inside python it should be cheap.

No, that's exactly the point I'm making, copying PTEs is not cheap on a large address space, woth many VMAs.

You can run a simple python script allocating a large list and see how it affects fork time.