Boring Problems Need Attention Too

Re:Boring Problems Need Attention Too by PaulHoule on Thursday June 17, 2021 @10:45AM (#27540131)

There is no such thing as a "word".

Look at the waveform of speech and you will see long silent gaps inside "words" as well as there frequently being no gap between "words".

There are phrases like "Skinny Puppy" that can do the same job as a word, there are also structures smaller than words that people smush together to make words. The two even work together:

   missile
   anti-missile missile
   anti-anti-missile missile missile

If you see "words" as the molecules of text there will always be an asymptote you can't overcome because segmenting text into words will sometimes introduce errors that you might not be able to recover from.

[ Parent | on HN ]

Re:Boring Problems Need Attention Too by belval on Thursday June 17, 2021 @10:54AM (#27540212)

One step further, I was doing text processing and my very western education taught me that characters were the smallest relevant construct when dealing with text.

Except that's simply not true for most eastern languages. Devanagari, Arabic, Bengla, Thai and so many other languages are packed with diacritics and graphemes, conditional rendering based on surrounding letters and much more that force you to completely rethink your approach.

[ Parent | on HN ]

Re:Boring Problems Need Attention Too by whatshisface on Thursday June 17, 2021 @01:57PM (#27542431)

Cursive writing in English has conditional rendering.

[ Parent | on HN ]

Re:Boring Problems Need Attention Too by lucasmullens on Thursday June 17, 2021 @12:51PM (#27541688)

I don't understand this. The article showed 4 text processors have a different definition of "word". How does that mean "there is no such thing"? Just because no one agrees on a definition for "word" doesn't mean words don't exist.

[ Parent | on HN ]

Re:Boring Problems Need Attention Too by mgbennet on Thursday June 17, 2021 @01:53PM (#27542381)

GP's point is more about how words are spoken. When we hear speech in a language we understand, we parse the sounds as a collection of distinct units with individual meaning we call "words". But the sound wave itself isn't actually segmented in this fashion, and the separation into words occurs in our perception and interpretation. But I agree, words definitely exist, just as interpretations of sound and written characters.

[ Parent | on HN ]

Re:Boring Problems Need Attention Too by zorked on Thursday June 17, 2021 @03:16PM (#27543375)

I don´t think it has anything to do with our interpretation: the illiterate have no concept of words. They come from writing systems and are not a part of languages.

German has very long "words" just because they decided not to add lots of spaces when creating a writing system for it. There is no particular reason why it's "Bundesverfassungsgerich" instead of "Bundes verfassungs gericht".

[ Parent | on HN ]