They could have guarded the building, though, even if they don't want the agents to engage outside of the building. Guarding the entrance should be feasible. If it isn't, Congress should not have meetings there.