Even in the Win9x days, you could always rename a binary that was in use and copy the new one in its place, at which point newly spawned processes will use the new one while old processes continue to be unaffected.

Quite frankly I find the Linux/Unix-style behaviour of being able to delete/overwrite an in-use file a bit disconcerting, but that might just be because I'm more used to the Windows way.