News for hackers. Stuff that matters.
Re:Show HN: Graphsignal – ML profiler to speed up training a... by p1esk on Monday July 04, 2022 @07:03PM (#31982602)
The problem with nccl is it reports combined bandwidth: nvlink (intranode) and network. I want to see the network traffic, for example to identify a network link bottleneck when changing model or pipeline parallelism configuration.

p.s. if you solve this I’ll become a paying customer.

Re:Show HN: Graphsignal – ML profiler to speed up training a... by dmitrim on Tuesday July 05, 2022 @09:31AM (#31987994)
Understand, we'll definitely think about the network part. Just in case it may help, if `nvidia-smi nvlink -gt d` is useful for you in this context then there is a related metric NVLink Throughput Rate to compare runs and monitor. At least you might get an idea whether/how internal links are utilized.
Re:Show HN: Graphsignal – ML profiler to speed up training a... by p1esk on Tuesday July 05, 2022 @01:16PM (#31990954)
Yes, I thought about it - in theory I can measure the total traffic with mpirun, then substract nvlink traffic (as measured by nvidia-smi) from it. However I'm not 100% sure that the nvlink traffic from nvidia-smi is the same as the nvlink traffic component of the mpirun. I'd prefer to measure internode traffic directly (e.g. using Mellanox tools) as a more reliable method.
Re:Show HN: Graphsignal – ML profiler to speed up training a... by dekhn on Monday July 04, 2022 @09:20PM (#31983318)
Yes, exactly this.