All those territories count their people as US citizens (except American Samoa they need to fix that), most have voted to either be part of the US or enjoy popular support of remaining so.

That's not what empires do.