Your tar example appear to Work On My Machine. Probably I don't have a funky enough filesystem. Of course if the filesystem itself doesn't support invalid unicode names, there isn't much the os or tools can do.

But hey, if enough tools and applications stop supporting non-utf-8 names, it is possible that in a a few decades you might get what you want as de-facto non-utf8 files will just disappear.