Sure, that downside is probably offset by the benefits of code reviews, but that was exactly what I said: receiving a review is beneficial, it's just a bummer that there's the downside that it's not instantaneous.
And if you structure your day well, you can do the rest of the context switching at a point when it's not a burden anyway - such as making changes in response to code review first thing in the morning before returning to your other tickets.
Again, in practice I still do actually switch to a different problem if that allows me to get my code reviewed. It's just that I wished it didn't come with the productivity hit that I experience.
For preliminary foundational work on a big project, where you’d have to change everything if the early work changes in response to code review, one good solution is to do an informal design review before jumping in to get feedback on plans for data model, architecture, boundaries, and APIs from a high level.