The form of this story has used widely since, and those stories have been incredibly popular with both authors and audiences for well over 100 years now. Implying that a tried-and-true technique of story-telling lies outside of storycraft betrays basic a lack of understanding of the subject.
Story tellers enjoy this form for the challenge of balancing what the omniscient narrator reveals-- too much or too strong in one direction and the whole thing falls flat. Readers enjoy the form because it feels a bit like ski jumping-- the writer provides the momentum and the reader's imagination leaps off the end with all the story's implications flying past.
In fact, Nabokov balanced this same form masterfully in the short story, "Signs and Symbols." The effect there is that the reader leaps off into a level of... well, I don't want to ruin it. :) But it's enough to say that outside of VR I don't think that story could have been effectively written without using the lady-or-the-tiger form.
Great metaphor. The story zips you down at full-speed, and then launches you upwards into the air. If you're comfortable feeling ungrounded, you'll be able to look across the horizon, in awe of the sights you couldn't see before. But if you're a lazy or cowardly person, your overriding urge will to return to the earth, to get back down to the safety of the soft slopes. Heck, later on, you might even petition the proprietor of the ski-resort to get rid of the ramp.
I think partly with this story it was because it was on the front page of HN, so I had very high expectations. The story is clearly building to something, to me in 2020 it seemed disappointing. I'm interested to hear how it was received in 1882 though.
This kind of device is used to great success in modern fiction, such as Stephen King's (non-supernatural!) short story "All that you love will be taken away", where the conclusion is not only open-ended, but also left to chance!
The story could, for example, have explored the issue of which way the princess might lean from the guys point of view, which covers all the same human nature angles, and then it could also have gone on to have a proper ending. I value storycraft.
I'm not saying the story should have been different back then, I'm sure it was perfect for 1882, and thats why its remembered now. I'm interested in how I differ from the 1882 audience.
edit: and koans are more about invoking mind states beyond words, like a kind of mind-hack, than they are about pondering ponderable things.