Ultra Fast Bert

Re:Ultra Fast Bert by marcinzm on Wednesday November 22, 2023 @08:47AM (#38379052)

Bottleneck for larger models however this would presumably allow for cheaper models at scale or on compute constrained devices (like phones).

[ Parent | on HN ]

Re:Ultra Fast Bert by entropicdrifter on Wednesday November 22, 2023 @01:09PM (#38382782)

And potentially for distributing a model across several devices at inference time. You could devote a cluster of smaller/weaker machines to inference.

[ Parent | on HN ]

Re:Ultra Fast Bert by sroussey on Wednesday November 22, 2023 @02:16PM (#38383767)

You can do that today, the only advantage today though is being able to fix the model in memory. It’s sequential and slower due to communication costs, though batching might be faster?

[ Parent | on HN ]