Re:Mass Editing Memory in a Transformer
by mrtranscendence on Friday April 21, 2023 @01:05PM
(#35656800)
When my wife tells me “you should call your mother”, I don’t think her brain assigned probabilities to “you should call your TV”, “you should call your xylophone”, “you should call your airplane”, etc, and then chose a suitable high-probability word (“mother”).
Would the natural analogy of “tokens” be “words”, or something more like, “portion of mouth-movement”?
I more imagine someone speaking another language, when I break down written phonemes for tokenization purposes.
As a native English speaker who grew up around code-switching Spanish speakers, I often heard speech that sounded really fast.
But I was hearing “hay-un-a-al-pa-ca-per-si-gui-en-do-un tren-de-car-ga”
I heard that as individual tokens, while my friends simply heard “¡There is an alpaca chasing that freight train!”
Talk so someone with aphasia or other brain disorder, or talk to a neurologist about it, and you'll see that's not quite the case. There really does seem to be a probability of saying TV when you meant to say "your mother", it's just that the language centers of our brains don't consciously calculate your probabilities and present them to our conscious mind.
See, my reaction has been, “perhaps our reasoning and actions are pretty much just a biologically-encoded statistical model too, it just doesn’t _feel_ that way because of some other factor.”