Re:Ziplm: Gzip-Backed Language Model
by dkatz23238 on Saturday July 15, 2023 @10:20AM
(#36737020)
What would happen if each word in "tokenized" to an integer and then you generate tokens instead of characters to produce a string of coherent words instead of random strings? Maybe the answer is obvious but not to me without diving into it at a deeper level. Would be interested to hear anyones thoughts on this.
[ Parent
| on HN ]
There are no comments on this comment yet.