I think we should discount The Widget Formerly Known As The Address Bar from these sorts of discussions. The omnibox[chrome]/awesomebar[ff]/frustrationmaker[edge]/etc. is a field where a human can type or paste all sorts of guff, and where the browser can present the human with a representation of the location of the cat video currently playing on the screen (complete with optional scheme, padlock icon, elided path elements, fancy colours and shading, Unicode snowmen, etc.)
Sometimes the guff from the human looks exactly the same as a URL, sometimes it's close enough to work out what they meant, and the rest of the time it's just guff. If it looks exactly like a URL or IRI, it's trivial for the browser to turn it into a proper URI and emit an appropriate request. If it's close, then some assumptions can be made, but the browser is still responsible for constructing the actual URI. That doesn't mean that what the human typed has any bearing on the underlying protocols or standards, or the definition of a URI/URL.
- Somewhere within some unstructured text: Here, finding the exact end of the URL is difficult and disallowing spaces can have a great benefit. Even then you can have ambiguities: Does punctuation at the end belong to the URL? What about URLs that are wrapped in <> brackets?
Whether or not it's even possible that the URL contains non-ascii chars depends on the type of the document.
- As a value of a HTML attribute without surrounding quotes: Slightly less ambiguous but including a space would break the HTML parser. The parser works fine with unicode characters however, so there is less pressure for authors to encode them.
- As a value of a quoted HTML or XML attribute, HTTP header or data field: Charset, Start and end of the URL are communicated out-of-band so there is no pressure at all to normalize the URL.
- In the browser address bar: Start and end are unambiguous but there are other concerns: You have to detect if it's an actual URL or a search term, you have to deal with omitted parts (e.g. no scheme) and people WILL input unicode characters. You also want to display unicode characters to the user, except when you don't want to for security reasons. And then users will copy the URL in the address bar and paste them into an HTML document...