Second, how would you solve priority inversion for general semaphores? With mutexes, you know what thread is holding the mutex, and can bump its priority correspondingly. With semaphores, there's no guarantee that the same thread that incremented the semaphore will decrement it later.
General semaphores? I'll easily grant that would be difficult. Not impossible, but likely wasteful and a PITA.
Binary semaphores, which is where the debate lies? Store a whoDecremented thread pointer, cleared on increment. Which is functionally identical to Dekker's algorithm[1] and Eisenberg and McGuire's algorithm[2], except they require storage for all possible waiting threads, instead of the single storage of a semaphore.
edit: links: [1]: http://en.wikipedia.org/wiki/Dekker%27s_algorithm [2]: http://en.wikipedia.org/wiki/Eisenberg_McGuire_algorithm