A developer upstream naively split on /n and leave some errant /r characters everywhere? Fixed.
Embedded backspaces or nulls or tabs or any sort of control character? Gone.
Non-ASCII data in a field you know should be ASCII only? Ha, not today good sir!
Until you've had to deal with the hell that is raw, free form data from who-knows-where, you cannot even fathom how satisfying it is to be able to deploy that regex (when appropriate) and know beyond a doubt that you can't have any more gremlins hiding in that particular field/data that'll hit you later.
In Portuguese, which I’ve worked with, you develop other tricks, like replacing à, á, â or ã with a. But, in order to do this, you still need to find out the encoding used before you can create the “ASCII” equivalent.
Fun trivia: coco means coconut; cocô means poo. So, by replacing ô with o, you’re guaranteed a chuckle at some stage.
Never did I see the parent mention "this is sufficient for humans", or just "...for English" - even that would be a naïve assumption (see? ;)).
I would rather pass around a link to https://ja.wikipedia.org/wiki/メインページ than to https://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3... , and lots of software will support both. And if you want all URLs to be ASCII, you'll need to convert the first into the second, not just delete the Japanese characters.
In user interaction with a browser or wherever else it seems that anything goes.
It's true that protocols such as HTTP only use ASCII URIs on the wire. If you are implementing an HTTP client yourself, you will need to implement percent-encoding.
Which is different from saying "well, I know URLs should be ASCII on the wire, so I can safely delete all the non-ASCII characters from a URL." That's not true.
If you look at the table at the top of [1], you'll notice all of the characters at the beginning of the ASCII range which are non-printing and therefore invisible. Plus, at the end for some insane reason, the DELETE character. If there's no valid reason for any of these characters to exist in your dataset, nor for higher code point (UTF) characters to exist, then [^ -~] will match them and let you strip them out all in one go.
Oh dear, I think I finally understand that mysterious phenomenon in some websites where, text I write in my native language gets saved and displayed as empty text, but writing in English works. I've come across this a few times over the years, in random forgotten corners of the Internet. It could well be that this regex got written down somewhere as good security practice, and some developers out there copied it into the code without thinking about whether the ASCII restriction is applicable.
That said, I do agree it's not something that should be used without thought. I mentioned that my usage generally involves knowledge that the data set has no valid reason to contain higher code points, and usually involves usage of Unidecode[2] to convert higher code points into ASCII equivalents (which in many cases strips out contextual knowledge, but is sometimes an acceptable trade off for stability and predictability of the data sent downstream).
[1] https://www.w3schools.com/html/html_charset.asp [2] https://pypi.python.org/pypi/Unidecode
[1] https://support.google.com/websearch/answer/2466433?hl=en
[2] https://www.google.com/search?hl=en&q="%0B%08+Foo+%7F"
[3] You can see the full string being searched here, it's 7 characters long: https://r12a.github.io/uniview/?charlist=%08%20Foo%20%7F
echo "lörem ipsum" | grep -P "[^ -~]" --color
and not
echo "lörem ipsum" | grep "[^ -~]" --color
I manage and source data for a lead generation/marketing firm, and dealing with free text data from all over the place is a nightmare. Even when working with billion dollar firms, data purchases take the form of CSV files with limited or no documentation on structure or formatting or encoding, sales-oriented data dictionaries, and FTP drops. I have a preprocessing script in python that strips out lines that can’t be parsed as utf8, stage it into a Redshift cluster, then hit it with a UDF in Redshift called kitchen_sink that encapsulates a lot of my text cleanup and validation heuristics like the above.