You can get off this GPU circus and simply go with purpose-built AI solutions.
You can buy single tensor accelerators from Google: https://www.coral.ai/products/
You can buy a bunch of those integrated into a single PCI-E card. https://iot.asus.com/products/AI-accelerator/AI-Accelerator-...
Cheap too. Some of these work with Mac. More of them work for PC, because the hardware interface is outside of Apple's thin vertical slice/garden.
These are devices for Tensorflow Lite which is more appropriate for IoT etc. not doing the intensive initial training of a complex model
Could be worth tracking what you did and make a new set of instructions, and trying to reproduce with a fresh install.
This is something Apple should pay people for.
Deep learning support for Mac is not going to happen at a level of quality you can rely on for research & dev work (like PyTorch + TensorFlow). The underlying problem is no big company cares about Mac platform and the work to maintain framework support for a specific piece of hardware is way beyond a hobby project. If you want your own on-prem hardware just buy Nvidia.
Now i've got a team of data scientists in a fully MBP shop and we're holding off upgrades to M1 until this all gets resolved.
On my personal M1, I managed to make it work, but its hard to know the layers of changes made and what exactly allowed it to work.