> the program is not aware of where you clicked. Fixing this would also require patching the world.

No, that exists for a long time and is widely implemented among terminal software. Demo: https://github.com/tinmarino/mouse_xterm