Glibc Buffer Overflow in Iconv

Re:Glibc Buffer Overflow in Iconv by tialaramex on Sunday April 21, 2024 @07:05AM (#40104695)

Of these outcomes I like the choice to pick nits best. I felt the same way before I learned any Rust (ie I wrote C where I had to manually groom strings) so this went in my pile of things to like about Rust. Apparently Rust only chose relatively narrowly to have &str (the string slice type, which is guaranteed to be UTF-8 text) at all, rather than just &[u8] (a slice of bytes) everywhere and to my mind that's a pretty serious benefit.

The choice to have this in a language with the safe/ unsafe distinction works very nicely because in so many languages you'd have this promised UTF-8 type and then in practice everybody and their dog uses the unsafe assumed conversion because it's easier, but in Rust you're pulled up short because that conversion needs an unsafe block, your local style may require (and good practice certainly does) that you address this with a safety comment explaining why it's OK and... it just isn't, in most cases. So you write the safe conversion instead unless you really need not to. This is a really nice nudge, you can do the Wrong Thing™, but it's just easier not to.

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by masfuerte on Sunday April 21, 2024 @07:18AM (#40104758)

Rust still has its warts. When dealing with an archive, say, you can find yourself needing to deal with Windows strings on Unix, or vice versa, but Rust only provides the string type for the platform you are running on. It could do with UnixString and WindowsString in addition to OsString.

MFC had a similar problem for years. It had a CString class which was ANSI or Unicode, depending on how your compiled your app, but moderately often you needed the other one, so it should have had a CStringA and CStringW too, with nice conversions between them.

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by tialaramex on Sunday April 21, 2024 @07:27AM (#40104800)

OsString isn't "the native string type", it's a container for whatever was convenient for Rust's internals on that OS, there are convenience functions so it probably feels like it's "UnixString" or "WindowsString" but it's neither.

In most of these file format cases what you've got is &[u8] or &[u16] and maybe it's the NonZero variant instead, so I think it's fine to be explicit that's what is going on and maybe in the process remind you to check - is this data UTF16LE? UTF16 with a BOM? UCS2 with a nod and a wink? Just arbitrary 16-bit integers and good luck?

But like I said, I favoured "picking nits" long before I learned Rust, so mileage may vary.

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by pornel on Sunday April 21, 2024 @10:08AM (#40105911)

IMHO Rust got OSString wrong – it indirectly promised that UTF-8 can always be cast to it without any copying or allocations, so on Windows it has to use WTF-8 rather than UCS2/UTF-16. Instead of being OS’s preferred string type, it’s merely a trick to preserve unpaired surrogates.

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by tialaramex on Sunday April 21, 2024 @11:55AM (#40106716)

Huh. Where is the "indirect promise" ? Is there a (conventionally hinted as "free", so it's not good if there's one which isn't very cheap) as function? Like as_os_string or something?

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by LegionMammal978 on Sunday April 21, 2024 @01:44PM (#40107613)

It's because str implements AsRef<OsStr> [0]. The function signature promises that whenever you have a borrowed &'a str, the standard library can give you a borrowed &'a OsStr with the same data.

Since references can't have destructors (they don't own the data like an OsString does), it means that the standard library can't give you a newly-allocated string without leaking it. Since obviously it isn't going to do that, the &OsStr must instead just act as a view into the underlying &str. And the conversion can't enforce any extra restrictions on the input string without breaking backward compatibility.

The overall effect is that whatever format OsStr uses, it has to be a superset of UTF-8.

[0] https://doc.rust-lang.org/1.77.2/std/ffi/struct.OsStr.html#i...

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by tialaramex on Sunday April 21, 2024 @03:27PM (#40108468)

Ah, an AsRef<OsStr>, yep that would do it. Thanks.

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by jcranmer on Sunday April 21, 2024 @02:26PM (#40107984)

I wouldn't say it's necessarily wrong; it may be (accidental) foresight. Windows has added a UTF-8 code page, which means you can get as near enough as makes no difference full support with the A functions instead of the W functions.

That said, even now in 2024, it's not clear how much of a bet Windows is making on UTF-8 versus UTF-16.

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by pornel on Sunday April 21, 2024 @02:53PM (#40108192)

I would be surprised if the UTF-8 support in Windows was anything deeper than the A functions creating a W string and calling W functions, which is what Rust is doing already.

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by tialaramex on Sunday April 21, 2024 @08:26PM (#40110444)

The question is whether that's all they ever do, or whether gradually some parts of Windows begin to decide to be UTF-8 first and their W functions now translate the opposite way. This must be tempting for some networking stuff, where you're otherwise taking a perf hit compared to Linux.

[ Parent | on HN ]

Re:Glibc Buffer Overflow in Iconv by pjmlp on Sunday April 21, 2024 @10:32AM (#40106050)

The first time I had to deal with this was on Turbo Pascal for Windows, where we got C strings alongside Pascal ones.

Then when 32 bit came to be we had the other variations on top.

Got to love leaky abstractions.

[ Parent | on HN ]