Well, I think asking a Gentoo guy if the compile times are worth the performance boost is quite valid (I used Gentoo for several years).

But just because you are using Linux, doesn't mean you have to spend a lot of time customizing. Actually, customizing can be a lot quicker on Linux than on Windows for example (due to the well-integrated package managers).

That said, I do think that some customizations help with productivity, but you should know your limits. If you are trying something completely new, which nobody has done before, you are unlikely to find huge productivity boosts. But if you cautiously follow some best practices you might find some productivity treasures.