Ok, well, "cargo build --release" took 3 minutes and 55 seconds, which I would consider reasonably fast for compiling and optimizing 300+ libraries. I guess your laptop has only 2 cores in an older processor, which would explain the difference. Kudos to you for keeping good hardware alive.

sometimes it takes 100x more without any apparent reason - it's probably because of memory or something, what takes minute to compile on my MBA, sometimes never finishes on raspi3 (swap is off)

I currently compile on raspi4 and scp the results there to check.