Upon thinking about this, it seems like it's a different problem in that the people are moving while the screen is not. So maybe the screen should have a camera and try to adjust the direction to the way the person is facing? Obviously if the screen is upsidedown that's not great.

(I don't think that anyone does this, but it's a neat thought.)

This would finally enable a window manager feature I have always wanted: "focus on the window I am looking at".

This sort of exists, actually... use hover focus and eViacam (https://eviacam.crea-si.com/) together. Of course, that doesn't give you separate control of the mouse, but I'm not sure that's a bug.

What happens when multiple people are looking at it from different directions, as if it was a table?

Then it has to start a pacman game! (I used to love those tables with the games in them that you can play from the person across from you)

I would suggest some manual switch (software or hardware), so that you can rotate displayed image as required, not yourself. Like a selector with 4 buttons, you click the one which is currently up for you and display respects that.