News for hackers. Stuff that matters.
Re:Ziplm: Gzip-Backed Language Model by dkatz23238 on Saturday July 15, 2023 @10:20AM (#36737020)
What would happen if each word in "tokenized" to an integer and then you generate tokens instead of characters to produce a string of coherent words instead of random strings? Maybe the answer is obvious but not to me without diving into it at a deeper level. Would be interested to hear anyones thoughts on this.

There are no comments on this comment yet.