If you are using buildx+QEMU for compiling your much better off cross-compiling inside the native architecture of github actions and the export the result to a build step that emulates the architecture: I went form cross compilation that took 2 hours to 10 minutes total

You can follow my Dockerfile of my project as an example on how to do it