So where do we draw the line? Of course, no real computer implementation of any language is going to be able to generate fundamentally pure code because in the end they're going to be mutating memory (at the very least the program counter).

That's not a very useful distinction, and usually one talks about "purity" at a conceptual level. The language provides a layer of abstraction in which things can be considered pure (or not), even if the underlying implementation e.g. makes optimizations that require mutation. The language is just a guarantee that you can ignore these aspects of its implementation.