News for hackers. Stuff that matters.

A comment on T0* – Series of encoder-decoder models trained on a large set of different tasks. Parent comment

Re:T0* – Series of encoder-decoder models trained on a large... by craffel on Monday October 18, 2021 @11:10AM (#28906197)
(author here)

The paper/model/code was just made public today. This may be why no one is talking about it yet.

Regarding whether the size is a hassle: It's possible to run inference on a single Google Cloud TPU v3-8 device or on a server with 4x 32GB v100 GPUs. Hugging Face also has an inference API for any model on the Hub: https://api-inference.huggingface.co/docs/python/html/index....

[ Parent | on HN ]
Re:T0* – Series of encoder-decoder models trained on a large... by NavinF on Monday October 18, 2021 @12:03PM (#28906973)
Do you have (rough) numbers for inference latency on 4x 32GB v100?
[ Parent | on HN ]
Re:T0* – Series of encoder-decoder models trained on a large... by VictorSh on Monday October 18, 2021 @06:49PM (#28912807)
(author here)

I don't have exact numbers for latency but the inference widget is currently on a TPU v3-8 (which if I am not mistaken could roughly be compared to a cluster of 8 V100). That gives you a rough idea of the latency for short inputs.

Note that a colleague just reminded me that it is possible on a single (big) GPU with enough CPU to run inference for T5-11B (which is the size we use) with offloading -> https://github.com/huggingface/transformers/issues/9996#issu...

[ Parent | on HN ]
Re:T0* – Series of encoder-decoder models trained on a large... by ourlordcaffeine on Monday October 18, 2021 @12:13PM (#28907129)
On the topic of GPT-3, I asked your creation:

"Who is better, you or GPT-3?"

> GPT-3

[ Parent | on HN ]
Re:T0* – Series of encoder-decoder models trained on a large... by ai_ia on Monday October 18, 2021 @05:49PM (#28912255)
It somehow picked up Modesty.
[ Parent | on HN ]
Re:T0* – Series of encoder-decoder models trained on a large... by echelon on Monday October 18, 2021 @11:46AM (#28906721)
Can this be used to generate prose at length? Or Reddit comment replies?
[ Parent | on HN ]
Re:T0* – Series of encoder-decoder models trained on a large... by srush on Monday October 18, 2021 @12:27PM (#28907322)
While in theory it could, the nature of its training favors shorter more factual replies.
[ Parent | on HN ]