News for hackers. Stuff that matters.
Re:Meta Unveils New AI Supercomputer by vl on Monday January 24, 2022 @03:36PM (#30063660)
Honestly, this single GPU-based install is child's play compared to Google's multiple TPU exoflop supercomputers with hyper-cube optical interconnects. Google's ML setups allow synchronous weight update on thousand+ TPUs...
Re:Meta Unveils New AI Supercomputer by rawtxapp on Monday January 24, 2022 @03:42PM (#30063740)
TPUs are amazing, but in my experience, debugging issues with them can be a bit tricky. Since nvidia's gpus are more common place (especially outside gcp), you can find a lot more information when you get stuck, it's also more battle tested, etc.
Re:Meta Unveils New AI Supercomputer by 6gvONxR4sf7o on Monday January 24, 2022 @05:51PM (#30065214)
For what it's worth, jax is helpful to me here. You can drop out of the jit to debug it as if it were numpy.

Of course that assumes your issues aren't with the jit itself or inside pmap, etc. That shit's hard.

Re:Meta Unveils New AI Supercomputer by alex_sf on Monday January 24, 2022 @06:13PM (#30065464)
Tbh I thought I was being trolled with 'hyper-cube optical interconnects'.
Re:Meta Unveils New AI Supercomputer by vl on Monday January 24, 2022 @07:04PM (#30065981)
Actually, you are right, I mistyped. Although hypercube interconnects exist, and were used, for example, in AS400, system in question uses hypertorus topology.
Re:Meta Unveils New AI Supercomputer by cm2012 on Monday January 24, 2022 @06:32PM (#30065685)
For what its worth, for attention based advertising (youtube and display, not search), FB targeting blows Google out of the water. Not sure why but its consistent across brands.
Re:Meta Unveils New AI Supercomputer by mchusma on Monday January 24, 2022 @08:56PM (#30066920)
I have seen this myself, by I'm unsure if it's just a "ad quality" thing. For example, I can target exact placements on YouTube for my exact niche, and broad Facebook matching will outperform. I have tried YouTube and display for months with nothing within an order of magnitude as effective as Facebook.
Re:Meta Unveils New AI Supercomputer by dekhn on Monday January 24, 2022 @07:10PM (#30066036)
for TPUv3 it's 2D torus, not hyper-cube, right? Not sure if TPUv4 topology is externally published, but IIRC hypercubes are basically never used any more.
Re:Meta Unveils New AI Supercomputer by vl on Tuesday January 25, 2022 @12:01AM (#30068265)
I mistyped, one version is 2D torus, next is 3D torus aka hypertorus.