That extra key is o in vim's visual mode. It moves the cursor to the other end of the selection
With the text below, your cursor is the pipe character. From the cursor position, in normal mode, press 'vjok'
line1
line2|
line3
In the rare case case when I miss the mark, I wouldn't mind pressing an extra key to move the cursor to the other end of the selection and expand from there.
And yeah I feel that dissatisfaction too.