I think in this case the PyTorch team also involved. They have already fixed some annoying bugs in the last two days (like matrix multiplication often failing because of buffer size mismatches).
As for BERT inference performance, I am not sure what kind of speedups you are expecting. The M1 Pro gives me 2.6 TFLOPs in single precision matrix multiplication of 768x768 matrices. M1 Pro GPU performance is supposed to be 5.3 TFLOPS (not sure, I haven’t benchmarked it).
Re:Running PyTorch on the M1 GPU
by machinekob on Saturday May 21, 2022 @08:28AM
(#31457279)
Ahh nvm I was thinking about m1 max (my brain is damaged by the m1 naming and i didnt saw information that you are using m1 pro gpu)
Did you get only 2.55x on BERT/other transformer vs CPU version?