When people are in a room you can have multiple voice conversations at once. When you are remote, you cannot. I know you can replace voice side channels with text, but this is a lossy proposition.

Turn-taking in general is better in person due to high bandwidth channels for nonverbal communication.

I say all of that as an autistic person who has trouble with all kinds of tacit communication (it may be that I have studied it so much in order to fit in that I really notice it), so yes there are downsides to being in person (I can pretend to make eye contact on a video), but I'm talking here about why the average developer or product person might have good reason to prefer in-person for a two pizza meeting.

Nice whiteboarding app affordances are great, but the low effort of onboarding to a physical whiteboard also encourages broader participation.