News for hackers. Stuff that matters.
Re:You can't just assume UTF-8 by kstrauser on Monday April 29, 2024 @06:34PM (#40204933)
If you give me a computer timestamp without a timezone, I can and will assume it's in UTC. It might not be, but if it's not and I process it as though it is, and the sender doesn't like the results, that's on them. I'm willing to spend approximately zero effort trying to guess what nonstandard thing they're trying to send me unless they're paying me or my company a whole lot of money, in which case I'll convert it to UTC upon import and continue on from there.

Same with UTF-8. Life's too short for bothering with anything else today. I'll deal with some weird janky encoding for the right price, but the first thing I'd do is convert it to UTF-8. Damned if I'm going to complicate the innards of my code with special case code paths for non-UTF-8.

If there were some inherent issue with UTF-8 that made it significantly worse than some other encoding for a given task, I'd be sympathetic to that explanation and wouldn't be such a pain in the neck about this. For instance, if it were the case that it did a bad job of encoding Mandarin or Urdu or Xhosa or Persian, and the people who use those languages strongly preferred to use something else, I'd understand. However, I've never heard a viable explanation for not using UTF-8 other than legacy software support, and if you want to continue to use something ancient and weird, it's on you to adapt it to the rest of the world because they're definitely not going to adapt the world to you.

Re:You can't just assume UTF-8 by hnick on Monday April 29, 2024 @08:17PM (#40205852)
It depends on the domain. If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else.

Unfortunately Google and many other companies have decided UTC is the only way, so this causes issues with ICS files that use that format sometimes when they are generating their helpful popups in the GMail inbox.

Re:You can't just assume UTF-8 by kstrauser on Monday April 29, 2024 @10:11PM (#40206561)
Those aren’t timestamps. They’re descriptions of how to select them.
Re:You can't just assume UTF-8 by hnick on Monday April 29, 2024 @10:48PM (#40206754)
Oh yes true. Somehow I missed that important detail.
Re:You can't just assume UTF-8 by cesarb on Tuesday April 30, 2024 @10:25AM (#40211258)
> If you are writing calendar software, it is legitimate to have "floating time" i.e. your medication reminder is at 7pm every day, regardless of time zone, travel, or anything else.

If you have to take medication (for instance, an antibiotic) every 24 hours, it must be taken at the same UTC hour, even if you took a train to a town in another timezone. Keeping the same local time even when the timezone changes would be wrong for that use case.

Re:You can't just assume UTF-8 by kstrauser on Tuesday April 30, 2024 @01:41PM (#40213695)
There are very few drugs where that's a requirement. Your kidneys and liver aren't smart enough to metabolize anything at precisely the same rate every day anyway.
Re:You can't just assume UTF-8 by hnick on Tuesday April 30, 2024 @07:26PM (#40217703)
If you're there for a while, you'll need to adapt anyway since your biorhythms will too. But there are plenty of other cases like a reminder to check something after dinner, or my standard wake-up alarm in the morning. Or if I plan to travel, book lunch at a nice place for 1pm, and put it in my calendar I just want it to be 1pm wherever I go, without caring about TZ changes.

Calendars, alarms, and reminders have some overlap here and floating time can be good for some cases.

Re:You can't just assume UTF-8 by kccqzy on Monday April 29, 2024 @06:48PM (#40205048)
> For instance, if it were the case that it did a bad job of encoding Mandarin

I don't know if you picked this example on purpose, but using UTF-8 to encode Chinese is 50% larger than the old encoding (GB2312). I remember people cared about this like twenty years ago. I don't know of anyone that still cares about this encoding inefficiency. Any compression algorithm is able to remove such encoding inefficiency while using negligible CPU to decompress.

Re:You can't just assume UTF-8 by kstrauser on Monday April 29, 2024 @07:01PM (#40205182)
That doesn't seem like the worst issue imaginable. I doubt there are too many cases where every byte counts, text uses a significant portion of the available space, and compression is unavailable or inefficient. If we were still cramming floppies full of text files destined for very slow computers, that'd be one thing. Web pages full of uncompressed text are still either so small that it's a moot point or so huge with JS, images, and fonts that the relative text size isn't that significant.

Which is all to say that you're right, but I can't imagine that it's more than a theoretical nuisance outside some extremely niche cases.

Re:You can't just assume UTF-8 by bawolff on Monday April 29, 2024 @07:22PM (#40205364)
> Web pages full of uncompressed text

Are basically non existent. Almost all modern web servers transparently compress html. Sending uncompressed text over the wire is extremely rare.

Re:You can't just assume UTF-8 by iraqmtpizza on Monday April 29, 2024 @07:35PM (#40205483)
They shouldn't be non-existent. Zip-then-encrypt is not secure due to information leakage.

EDIT: also, it's not safe—message length is dependent on the values of the plaintext bytes, period. i'm not saying don't live dangerously, i'm just saying live dangerously knowing

Re:You can't just assume UTF-8 by fanf2 on Monday April 29, 2024 @07:48PM (#40205617)
The information leakage problem occurs when compression is done in the TLS layer, because then the compression context includes both headers (with cookies) and bodies (containing potentially attacker-controlled data). But if you do compression at the HTTP layer using its Transfer-Encoding then the compression context only covers the body, which is safe.
Re:You can't just assume UTF-8 by rocqua on Tuesday April 30, 2024 @04:04PM (#40215650)
It can still leak data if attackers can get their input reflected. I.e. I send you a word, and then I get to observe a compressed and encrypted message including my word and sensitive data. If my word matches the sensitive data, the cyphertext will be smaller. Hence I can learn things about the cipgertext. That is no longer good encryption.
Re:You can't just assume UTF-8 by kstrauser on Monday April 29, 2024 @07:57PM (#40205692)
Do you often send raw bitmaps for the same reason?
Re:You can't just assume UTF-8 by iraqmtpizza on Tuesday April 30, 2024 @02:32AM (#40207866)
Do you often get completely pwned and have your encrypted calls transcribed by people eating doughnuts because you thought it was safe to compress sensitive data before encrypting? https://web.archive.org/web/20080901185111/https://technolog...
Re:You can't just assume UTF-8 by bawolff on Monday April 29, 2024 @09:54PM (#40206453)
What you are talking about is generally referred to as the "BREACH" attack. While there may theoretically be scenarios where it is relavent, in practise it almost never is so the industry has largely decided to ignore it (its important to distinguish this from the CRIME attack which is about http headers instead of the response body which has a much higher liklihood of being exploitable while still being hard).

The reason its usually safe is that to exploit you need:

- a secret inside the html file

- the secret has to stay constant and cannot change (since it is adaptive attack. CSRF tokens and similar things usually change on every request so cannot be attacked)

- the attacker has to have a method to inject something into the html file and repeat it for different payloads

- the attacker has to be able to see how many bytes the response is (or some other side channel)

- the attacker is not one of the ends of the communication (no point to attack yourself)

Having all these requirements met is very unlikely.

Re:You can't just assume UTF-8 by PeterisP on Monday April 29, 2024 @06:53PM (#40205087)
A key aspect is that nowadays we rarely encode pure text - while other encodings are more efficient for encoding pure Mandarin, nowadays a "Mandarin document" may be an HTML or JSON or XML file where less than half of the characters are from CJK codespace, and the rest come from all the formatting overhead which is in the 7-bit ASCII range, and UTF-8 works great for such combined content.
Re:You can't just assume UTF-8 by tomoyoirl on Monday April 29, 2024 @07:01PM (#40205183)
To say nothing of mod_gzip
Re:You can't just assume UTF-8 by gary_0 on Monday April 29, 2024 @07:31PM (#40205456)
For Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so in practice there is no data size advantage to using non-standard encodings.

[0] https://utf8everywhere.org/#asian

Re:You can't just assume UTF-8 by neild on Monday April 29, 2024 @09:36PM (#40206363)
In addition, Chinese characters encode more information than English letters, so a text written in Chinese will generally consume fewer bytes than the same text in English even when using UTF-8.

(Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)

Re:You can't just assume UTF-8 by kstrauser on Monday April 29, 2024 @10:14PM (#40206570)
True, but even then you wouldn’t want to store it egregiously badly.
Re:You can't just assume UTF-8 by Panzer04 on Tuesday April 30, 2024 @01:17AM (#40207474)
Presumably that derives from the overhead of encoding an english character as a full byte? Given there's only 26 characters normally, you could fit that into 5 bits instead, which funnily enough does actually line up with the chinese character encoding (5x5 vs 1x24).
Re:You can't just assume UTF-8 by kps on Tuesday April 30, 2024 @11:03AM (#40211767)
Yes. It's the non-Latin alphabets that lose with either UTF-8 or UTF-16, compared with stateful ISO 2022 page switching.
Re:You can't just assume UTF-8 by LaffertyDev on Monday April 29, 2024 @11:17PM (#40206888)
I can't help myself. The grandest of nitpicks is coming your way. I'm sorry.

> If you give me a computer timestamp without a timezone, I can and will assume it's in UTC.

Do you mean, give you an _offset_? `2024-04-29T14:03:06.0000-8:00` the `-8:00` is an offset. It only tells you what time this stamp occurred relative to standard time. It does not tell you anything about the region or zone itself. While I have consumed APIs that give me the timezone context as part of the response, none of them are part of the timestamp itself.

The only time you should assume a timestamp is UTC is if it has the `z` at the end (assuming 8601) or is otherwise marked as UTC. Without that, you have absolutely no information about where or when the time has occurred -- it is local time. And if your software assumes a local timestamp is UTC, then I argue it is not the sender of that timestamp's problem that your software is broken.

My desire to meet you at 4pm has no bearing on if the DST switchover has happened, or my government decides to change the timezone rules, or if {any other way the offset for a zone can change for future or past times}. My reminder to take my medicine at 7pm is not centered on UTC or my physical location on the planet. Its just at 7pm. Every day. If I go from New York to Paris then no, I do not want your software to tell me my medicine is actually supposed to be at Midnight. Its 7pm.

But, assuming you aren't doing any future scheduling, calendar appointments, bookings, ticket sales, transportation departure, human-centric logs, or any of the other ways Local Time is incredibly useful -- ignore away.

Re:You can't just assume UTF-8 by Spivak on Monday April 29, 2024 @11:55PM (#40207071)
It's always nice to see someone who actually understands time.

"Convert to UTC and then throw away the time zone" only works when you need to record a specific moment in time so it's crazy how often it's recommended as the universal solution. It really isn't that hard to store (datetime, zone) and now you're not throwing away information if you ever need to do date math.

Re:You can't just assume UTF-8 by globular-toast on Tuesday April 30, 2024 @03:11AM (#40208078)
Yeah, I've been trying to convince people forever to store time zones with timestamps when appropriate. If you record events from around the world and don't record what time zone they happened in you can't even answer basic questions like "what proportion happened before lunch time?"

People love simple rules and they will absolutely take things too far. Most developers learn "just use UTC!" and think that's the last thing they ever need to learn about time.

Re:You can't just assume UTF-8 by kstrauser on Tuesday April 30, 2024 @12:39AM (#40207299)
As I mentioned in another reply, "remind me every day at 7PM" isn't a timestamp. It's a formula for how to determine when the next timestamp is going to occur. Even those examples are too narrow, because it's really closer to "remind me the next time you notice that it's after 7PM wherever I happen to be, including if that's when I cross a time zone and jump instantly from 6:30PM to 7:30PM".

Consider my statement more in the context of logs of past events. The only time you can reasonably assume a given file is in a particular non-UTC TZ is when it came from a person sitting in your same city, from data they collected manually, and you're confident that person isn't a time geek who uses UTC for everything. Otherwise there's no other sane default when lacking TZ/offset data. (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).

Re:You can't just assume UTF-8 by LaffertyDev on Tuesday April 30, 2024 @11:03AM (#40211776)
> As I mentioned in another reply, "remind me every day at 7PM" isn't a timestamp. It's a formula for how to determine when the next timestamp is going to occur. Even those examples are too narrow, because it's really closer to "remind me the next time you notice that it's after 7PM wherever I happen to be, including if that's when I cross a time zone and jump instantly from 6:30PM to 7:30PM".

That's certainly fair in the context of a recurring event with some formula. I caution that a lot of people will still immediately reach for timestamps to calculate that formula, particularly for a next occurrence, and in the context if this conversation they would be given as an ISO8601 datetime based on Local Time. I would also caution that calendar events that have a distinct moment in time that they start are also prime for Local Time where a UTC-default mentality will cause errors.

> Consider my statement more in the context of logs of past events

From the stance of computer generated historical log data, I definitely agree that UTC everywhere is a sane default and safe to assume :)

(And, in your defense, I would definitely argue UTC-everywhere gets you 95% of the way there for 5% of the effort... I get why people make the tradeoff)

> (I know they're not the same, but they're similar in the sense that they can let you convert timestamps from one TZ to another).

More nitpicking on my part, again, I'm sorry, it lets you convert from one _offset_ to another or from an offset to UTC. Think Arizona being a special snowflake who (mostly!) doesn't observe DST. You can't assume all UTC-7 offsets are all Mountain Time.

Re:You can't just assume UTF-8 by logrot on Tuesday April 30, 2024 @04:37AM (#40208585)
Fundamentally I agree, but sadly the world isn't that simple.

You usually end up with having to deal with whatever eccentric sh!t that ultimately comes from the same source as the payment for the job.

Re:You can't just assume UTF-8 by layer8 on Tuesday April 30, 2024 @07:45AM (#40209925)
> For instance, if it were the case that it did a bad job of encoding Mandarin

Please look up the issues caused by Han unification in Unicode. It’s an important reason why the Chinese and Japanese encodings are still used in their respective territories.

Re:You can't just assume UTF-8 by Shorel on Wednesday May 01, 2024 @05:54AM (#40221278)
That makes Unicode a failure then