You can see my early videos were crude meme attempts, and eventually it morphed to 100 TPU training. You can’t really predict the things that you’ll like, so don’t try.
What’s fun to you? That’s the question to focus on. For me, it was language modeling and image generation, which I learned from https://gwern.net/GPT-2 and Peter Baylor’s’ stylegan notebook, respectively. Then I transitioned to audio memes https://youtu.be/koU3L7WBz_s and kept going from there.
Tensorflow is bullshit, pytorch is different bullshit, Jax is pretty friggin awesome but still has one or two flecks of bs. But none of it feels hard to deal with, because it’s all an enormous box of legos; as long as you chase your interests, you’ll never* feel like it’s annoying.
* you’ll be annoyed all the time, but at least you’ll keep going.
The ray solution here is so simple. It was a joy to use. But I don't know anything about Dask.
By the way, if anyone wants to see how the loss graphs turned out: https://twitter.com/theshawwn/status/1406171487988498433
(I wish I'd uploaded the logs to tensorboard.dev for posterity. But you can see quite clearly in the screenshots all the information you'd want to see anyway, with apologies to blind engineers. Oh lord, is there a single blind engineer working in ML? Suddenly it's an appealing idea to try to make tensorboard accessible... I wonder how blind people could interpret graphs. Averages, probably.)
I don't remember what's normal for direct GPU<>GPU, but for many cases we see, the occasions we've done it is through a special pinned memory mode through a staging area. That used to be hard, but nowadays with the rapids.ai ecosystem (cupy / rmm / etc), nice python wrappers.
Dask is part of that ecosystem ("dask-cudf"), but helps more w/ automation around bigger-than-memory paging, multi-gpu dispatch, and multi-node dispatch. Underneath, it does some nice things for you, like setting CPU<>GPU affinities. However, doing custom peer-to-peer / NUMA stuff quickly gets you back to cupy/rmm, and thankfully, that's Python and integrates nicely with pandas/arrow/ etc :)
EDIT: There's a wild world of fancy NVLink GPU<>GPU interconnects in more expensive multigpu boxes. We mostly deal with more end-to-end IO issues like network/SSD array ->PCI cards->GPUs as the I/O bottleneck, such as during bigger-than-memory and oneshot/cold use, so I can't speak to the p2p bits as much.
My goal was to explore the extent of distributed computing using local low power compute nodes where different architectures are common and not to be compared with professional work like gp has detailed with homogeneous architectures.
But in case you'd like to indulge in similar masochist activities, I have couple of gists like installing Ray on ARM[1], Apache Arrow on ARM[2].
[1] https://gist.github.com/heavyinfo/aa0bf2feb02aedb3b38eef203b...
[2] https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603...
It scaled fine up to 33 TPUs (i.e. 33 layers).
Ray is impressive as hell.
By the way, I didn't write the code to do any of that. kindiana, aka "the guy that wrote GPT-J", also happened to write this:
https://github.com/kingoflolz/swarm-jax
I just ran it and it worked. Which is extraordinarily unusual for TPUs, historically speaking.
I'm pushing my luck at this point, since it's crossing the line from enthusiasm to spam. But if you want to try out Ray on TPUs like I did here, I posted a (massive) amount of detail on how to get started, and why: https://news.ycombinator.com/item?id=27728225
That's the last I'll be mentioning it for some time, though.
Ray + JAX is such a killer combo.