One way they might know is by seeing people posting online about how they used VPNs to get around the block. Crypto traders are often bad at being subtle.
It also alleges that they solicited American customers, which presumably they also have evidence of. For instance, maybe advertising on American websites or posting on forums they knew were frequented by Americans, etc. Maybe they posted winking messages that told Americans how to get around the IP ban? All sorts of possibilities.
Edit: from the indictment, page 7: "internal BitMEX records reflected thousands of BitMEX accounts with United States location information that were enabled for trading." So people volunteered their location information to BitMEX.
Page 12: "the defendants, knew that specific customers, residing in the United States, continued to access BitMEX' s platform... DELO and DWYER knowingly allowed one of these customers to access BitMEX using a non-U. S. passport in the name of a third party that did not belong to this customer. DELO also allowed another customer to continue to access a BitMEX trading account despite this customer being "US based," because "[h]e's famous in Bitcoin," and falsely changed this customer' s internal country of residence to a country other than the United States."
Less trivially: other, out-of-band clues based on support emails sent to BitMEX; location information in BitMEX profile; same username on other platforms and mention of location; that sort of thing