This is very interesting, thanks for sharing. Pretty nice writeup as well. I've been running data centers full of over 100k+ GPUs for the last few years.

I can't imagine involving k8s in that though. It just seems like a huge amount of complexity that isn't really necessary. But maybe that is just me... I really like keeping things as simple as possible.

Anyway, maybe I'll have to give it a deeper look... I'm sure I can learn something from it. =)