Spotify pays a discounted royalty rate when they can kick you over to random plays. Business over user experience, for sure, although they might say you wouldn't pay them a higher monthly subscription to compensate if they didn't do this.

Can't say why or how they arrive at such a tight list of random options; perhaps it's an attempt to give you something predictable while fulfilling royalty obligations. Maybe it's just broken.