Re:No Language Left Behind
by Vetch on Wednesday July 06, 2022 @10:32PM
(#32009352)
Hi, I'm looking but can't seem to find instructions on how to do tokenization. Where is spm model, is it "flores200_sacrebleu_tokenizer_spm.model" or something else? And is it direct or spm -> dict? Or how to prime model for a specific language pair?
[ Parent
| on HN ]
Re:No Language Left Behind
by hello_im_angela on Wednesday July 06, 2022 @10:54PM
(#32009465)
We tokenize with the flores-200 spm model, correct. To generate from the model, check out the instructions here: https://github.com/facebookresearch/fairseq/tree/nllb/exampl...