News for hackers. Stuff that matters.

A comment on Large language models are having their Stable Diffusion moment. Parent comment

Re:Large language models are having their Stable Diffusion m... by teruakohatu on Saturday March 11, 2023 @08:31PM (#35115286)
Longer inputs consume more memory, and if your input are shorter than the token length, they have to be padded.

So asking a simple one sentence question to a model that has a 81,920 token limit would be a collosal waste of resources.

[ Parent | on HN ]
Re:Large language models are having their Stable Diffusion m... by visarga on Sunday March 12, 2023 @01:36AM (#35117413)
Padding + masking. So the transformer doesn't waste time on the padding.
[ Parent | on HN ]