Could it be something as simple as your brother making eye contact hand displaying “I’d like to talk” micro expressions whereas you might look away or display “I’d really rather keep walking” micro expressions?

I know I’m in the latter camp, but if I force it I can spend some time in the former.