The difference, as I've seen it, is that mutexes more often have code to account for lock escalations and shared locks, and sometimes other potential problems, which are deemed unimportant for semaphores because they're used where such things are not needed. It's entirely possible to create a semaphore which does account for these - trivial, even, to make a binary semaphore have everything a mutex has. Similarly, a mutex does not need to have such capabilities to be considered a mutex.
All of which is simply meta-behaviors, not things which are part of their definitions. Debating about current implementations is useful only in terms of current implementations, not in defining something's inherent capabilities.
Second, how would you solve priority inversion for general semaphores? With mutexes, you know what thread is holding the mutex, and can bump its priority correspondingly. With semaphores, there's no guarantee that the same thread that incremented the semaphore will decrement it later.
General semaphores? I'll easily grant that would be difficult. Not impossible, but likely wasteful and a PITA.
Binary semaphores, which is where the debate lies? Store a whoDecremented thread pointer, cleared on increment. Which is functionally identical to Dekker's algorithm[1] and Eisenberg and McGuire's algorithm[2], except they require storage for all possible waiting threads, instead of the single storage of a semaphore.
edit: links: [1]: http://en.wikipedia.org/wiki/Dekker%27s_algorithm [2]: http://en.wikipedia.org/wiki/Eisenberg_McGuire_algorithm