News for hackers. Stuff that matters.
Re:Pile-T5 by gwern on Monday April 15, 2024 @12:32PM (#40042729)
Now that's a blast from the past - T5! I always thought it was underused, but it's also from so long ago now, and even Google has moved past it to UL2 etc AFAIK. What's the use-case here for reproducing it? Aren't there already many good code models?
Re:Pile-T5 by brizii on Monday April 15, 2024 @12:42PM (#40042852)
my group is currently working on a T5 model (and tokenizer) for html, as there are very few (if any) tokenizers that work well with HTML!

You can try using GPT4's tokenizer on your own HTML inputs below [1] ... there's definitely room for improvement!

[1] https://tiktokenizer.vercel.app

Re:Pile-T5 by dartos on Monday April 15, 2024 @12:46PM (#40042902)
Blast from the past? Like 2 years ago?
Re:Pile-T5 by p1esk on Monday April 15, 2024 @01:28PM (#40043337)
T5 was released in 2019.
Re:Pile-T5 by stavros on Monday April 15, 2024 @01:43PM (#40043511)
Isn't that two years ago?
Re:Pile-T5 by littlestymaar on Monday April 15, 2024 @01:50PM (#40043585)
Two years already? This locked-down year of 2020 really messes up with me perception of time.
Re:Pile-T5 by michaelt on Monday April 15, 2024 @01:09PM (#40043128)
Gather round, children, and let Grandpa tell you a tale from way back in the day, when we thought 24GB was a lot of VRAM...
Re:Pile-T5 by bevekspldnw on Monday April 15, 2024 @01:29PM (#40043350)
My nearly new 48GB A6000 is basically an ancient artifact relative to the exploding size of MoE requirements.
Re:Pile-T5 by emadm on Monday April 15, 2024 @12:40PM (#40042827)
I figured it would be interesting to see scale and for the image models
Re:Pile-T5 by euclaise on Monday April 15, 2024 @06:19PM (#40046350)
A lot of embedding models are built on top of T5's encoder, this offers a new option

The modularity of the enc-dec approach is useful - you can insert additional models in between (e.g. A diffusion model), you can use different encoders for different modalities, etc

Re:Pile-T5 by jakderrida on Wednesday April 17, 2024 @06:31PM (#40070766)
I've downloaded the data for every model on Huggingface and, if nothing else, flan-t5-xxl is the largest encoder (discriminant) model in terms of total file sizes. To be clear T5 is an encoder-decoder model. But the shift in focus to decoder-only has made it so encoder and encoder-decoder models have been, for lack of a better word, neglected, in terms of training focus.