My build times aren't as long, because I'm working with Java-based microservices, so each service takes a minute or so to build (with tests) but even that delay can break concentration. Turning off tests helps, but then you don't always have the immediate feedback of the test results (and don't worry, I always run the tests before committing).
Android is probably the only platform that in almost every conference has at least one talk related to how to improve build speed, thanks Gradle + Android Studio.
But now that I am working across projects that run 2-3 repositories at a time with Jetbrains IDE, the computer almost barely keep up. I get times where typing will actually lag.
I am contemplating asking them for a Mac Mini instead since I rarely ever venture out from my desk.
In native app development, Android Studio and Xcode are severely bottlenecked by 2 cores. The difference between my laptop and my Mac Mini (6 cores) is astounding.
The difference between a build taking 6x minutes and 1x minutes is more than just 1/6 because it goes from "Well, guess I'll check email/messenger/HN real quick." to taking a sip of coffee.
It also gets real bad when your IDE is forced to fight with your browser, email client, etc. Then you're forced to do nothing, lest your casual actions further slow down your local build.
Usually the "breaking point" is when builds take more than 5 minutes, but especially when they take more 30+ mins.