I don't think the the two endpoints are that big of a deal. I know where I want to start selecting, so I can just navigate there and then expand the selection in one direction only. This is how it works with vi anyway, only you can't really expand the selection after the fact.
In the rare case case when I miss the mark, I wouldn't mind pressing an extra key to move the cursor to the other end of the selection and expand from there.
And yeah I feel that dissatisfaction too.
That extra key is o in vim's visual mode. It moves the cursor to the other end of the selection
With the text below, your cursor is the pipe character. From the cursor position, in normal mode, press 'vjok'
line1
line2|
line3
When you make a motion you have to decide which one of the endpoints is the current cursor, and that's where I feel like you run into trouble. "Grow the selection up by one line" is a hard notion to express when you started by selecting down. Vim has visual select mode, which has a line-based, character-based, and block-based mode. I'm not totally sure that we need them to be distinct.
The problem is that kakoune now has me vaguely dissatisfied with both vim and kakoune and I don't like it.