This is definitely a feature I wanted to have in my own attempt (hopefully I can start work on it before the end of the year; I think I'll bump up my mental priority for blogging about the design). I also have considered symlink (not sure if this really works) and .pth file-based approaches.
(TIL `os.link` has been supported on Windows since 3.2.)
The reason uv is fast is that it creates hard links from each of your virtual environments to a single shared cached copy of the dependencies (using copy-on-write in case you want to edit them).
This means that if you have 100 projects on your machine that all use PyTorch you still only have one copy of PyTorch!