FWIW, I recently switched from an Intel to an M1 Macbook, and found shared volumes with Docker Desktop were unusably slow (previously, I'd used VirtualBox host docker and shared files into it via NFS).

However, using https://mutagen.io/ to keep the container in sync has worked remarkably well - it's greatly improved the speed of the container.