Re:Show HN: u8p – A Go Utility for Precise UTF-8 String Trun...
by catatsuy on Saturday May 04, 2024 @07:30AM
(#40256823)
Thank you for your reply.
I was thinking about whether to return an error. If we can’t find a UTF-8 start byte in the nearby 4 bytes, it’s unclear what to return. I thought maybe we could ignore this problem.
I don’t return the string itself because I don’t know if users want the start or the end of the string. Also, I want to avoid copying large strings. It’s up to the users how they use this function.
Since no one is using this package yet, we might consider changing the interface.
Re:Show HN: u8p – A Go Utility for Precise UTF-8 String Trun...
by lifthrasiir on Saturday May 04, 2024 @08:04PM
(#40261325)
> I was thinking about whether to return an error. If we can’t find a UTF-8 start byte in the nearby 4 bytes, it’s unclear what to return. I thought maybe we could ignore this problem.
This highly depends on the use case, and your stated use case doesn't seem to need any sort of error.
> Also, I want to avoid copying large strings.
Go strings are not copied in that way; they are implemented like immutable slices [1].
1. `LimitPrefix(a, n)` should always return some prefix of `a`, namely `a[:m]` where 0 < m <= n.
1a. In particular, `m` can be zero if `n` is small enough.
1b. And `m` is expected to equal to `len(a)` if `len(a) >= n`.
2. `a[:m]` should of course be a valid UTF-8 string if `a` already was.
3. `m` should be maximized under these conditions.
There are some edge cases that we have to fill in as well:
4. The conditions 1 and 1b are a reasonable expectation even for non-UTF-8 inputs. They are also easy to guarantee.
5. The condition 2 can't be efficiently extended for non-UTF-8 inputs and no justifiable use cases exist.
6. However the condition 3 depends on the condition 2 (and 1 of course). Therefore it should be replaced with something concrete, otherwise we risk an unintentional incompatibility.
7. Negative n may arise from a size calculation with a missing bound check, so treating it as zero sounds fair.
The current design, in comparison, just finds the last UTF-8 lead byte within `a[:l]`. It doesn't even help with the truncation: both `Find("thirty", 5)` and `Find("dreißig", 5)` return 4, but `"thirty"[:5]` is valid while `"dreißig"[:5]` is invalid. Also `Find("one", 5)` unexpectedly fails! An arbitrary condition of `l <= 3` is even more confusing.
---
Based on aforementioned conditions, I propose the following instead (warning: never tested):