Re:What we learned from using GPT for 500k+ classifications
by espe on Wednesday July 12, 2023 @06:26AM
(#36692536)
thanks for the clarification. if your base population is that large then it's frequencies and you get a fingerprint. well done.
Re:What we learned from using GPT for 500k+ classifications
by foolswisdom on Wednesday July 12, 2023 @08:26AM
(#36693544)
I'm curious though, how do you handle related forms of a word (assuming you don't use stemming)? It doesn't seem to me that this process would automatically handle that.
Re:What we learned from using GPT for 500k+ classifications
by espe on Wednesday July 12, 2023 @02:08PM
(#36699071)
i think they don't. given enough data those related forms will have similar colocation profiles (aka fingerprints). not a model of semantics.. at scale it might become a proxy though.
Re:What we learned from using GPT for 500k+ classifications
by 19h on Wednesday July 12, 2023 @02:49PM
(#36699831)
Yes, we don't actively cover this.
Our training dataset is composed of 12 different languages. The English corpus has a size of around 25TB -- ~18TB after a rather aggressive filtering process (char frequencies [i.e. https://gist.github.com/19h/b983769b7ec9c4a9528377d7819892a7], too many unusual features in the sentences, too short sentences, etc.).
Most common misspellings of words have nearly identical fingerprints, and we align the retinas across languages so that translated sentences result in similar fingerprints (via competitive learning).
Re:What we learned from using GPT for 500k+ classifications
by espe on Wednesday July 12, 2023 @04:11PM
(#36700968)
"retina" describes the orientation of the vector space? first time i hear it in this context.
Re:What we learned from using GPT for 500k+ classifications
by 19h on Wednesday July 12, 2023 @07:08PM
(#36702902)
Apologies, that's a very semantic folding specific term -- with retina I refer to a word -> fingerprint database that gives you either fingerprints of single words or of blocks of text (by applying relevant heuristics to filter out irrelevant noise / heavily used words in a specific language).
We may only be using 16k vector values but that doesn’t mean we only have a vocab of 16k —- our vocab is more around 1.9 million words each described by a sparse fingerprint of 16k.