> So Nancy Pelosi's office is generally open to the public, and it's fine for people to go on in whenever they want, even when she's not there?

Generally, yes. Their offices are suites with a reception that'll be staffed for constituent services most of the day. Staff are also typically there all day taking calls from the public.

Found a quick tour of one (small office; leadership will have much larger ones) on YouTube: https://www.youtube.com/watch?v=UnIMUfF3U8I