News for hackers. Stuff that matters.
Re:On a Great Interview Question by bluepizza on Thursday April 13, 2023 @09:35AM (#35555228)
This problem was solved by medieval monks by introducing white spaces.
Re:On a Great Interview Question by tetromino_ on Thursday April 13, 2023 @09:48AM (#35555420)
Many languages still don't put whitespace between some or all words (Chinese, Japanese). Many languages tend to use long compound words which are better represented as sequences of tokens rather than one token (German, Russian). Many inputs are dirty (coming from OCR, for example) so whitespace can be unreliable. And even in clean English text, inflected words may be better tokenized as sequences of tokens rather than a single token.