An arbitrary direction takes too many bits to encode, rather than a few set predicted directions. Reminder that you don't have to nail the prediction exactly, but choose a prediction that gives you the lowest residue error. So you need a way of encoding predictions that's smaller than your coded residue.