Yes, a well-designed system can minimize the impact by for example lazy-load the font file or even `Font Face Observer`[0] them for manual hot swap.
But that is not the main point. The main point is: You are the one who's in charge of the user experience, not Google. Don't shift the responsibility to somebody (external service) you don't have control of.
The software works the best when all it's parts are predicable, and make it more self-sufficient can help.
Google’s own pages don’t do that, they intentionally hide the text until the font loads to prevent a “flash of unstylised content.” (It’s quite a bit longer than a flash on many connections, plenty long enough to start reading.)
Not quite how web fonts work, usually they just swap in for the system fonts once loaded.