Maybe a mitigating factor is that any Clojure 2-arg function extends AFunction which implements Comparator. So any Clojure function that returns -1/0/1 will work transparently. Example, use - as a comparator:

=> (sorted-set-by - 2 9 3 5 4) #{2 3 4 5 9}