But not everybody can go and be slavemasters, so if you're going to pick arbitrarily who can do it, why not use race in the criteria?

Everybody can know how atoms work, and read Cicero, and figure out where on the shore a boat traveling diagonally will land if the wind is picking up at a given rate. Increasing the human knowledge pie in that way is what universities claim to do.