There's no government restrictions preventing you from assembling elsewhere.

Your interpretation would make shutting down any place where people assembled unconstitutional which was clearly never the intent.