News for hackers. Stuff that matters.
Re:You can't just assume UTF-8 by bobmcnamara on Monday April 29, 2024 @10:16PM (#40206579)
Heckin no thank you!

The FCC ULS database records are stored in a combination of no fewer than three different encodings(1252, UTF8, and something else for a handful of German names) that vary per record.

When I brought this up they said something to the effect of: it's already unicodes it has tilde letters!

Re:You can't just assume UTF-8 by eythian on Tuesday April 30, 2024 @10:32AM (#40211329)
I did once have a file that had UTF8, Windows-1252, MARC8, and VT100 (really) all mixed up in it. I think the data had gone through multiple migrations between software in its past.

I had write to my own "clean this as well as possible" thing, and it did a good enough job.