Using the Set interface is probably the right thing. Then you profile, and if the set implementation you chose is too slow, you just swap it out for another implementation that implements the Set interface.