You clearly doing something wrong as I get about 3 minutes per image on m1 mac mini.

But yeah, at this stage most of guides are early hacks and require individual tweaking. It is quite expected that people get varying results. I assume in a week or a month situation will get much better and much more user-friendly.