The caveat is that you lose whatever taxes you paid on exercising unvested shares. Gotta be really careful with that, as the taxes could cost way more than the exercise itself.

The idea with early exercising is to not pay any taxes because you exercise at the strike price.

Sure, but there are instances when that's not the case, e.g. early exercise implemented by company after the fact, or holding off buying until you're reasonably sure that the shares won't tank next month.