News for hackers. Stuff that matters.
Re:How to calculate the alignment between BERT and spaCy tokens by cochne on Monday July 12, 2021 @10:37PM (#27817156)
This is exactly why tokenizers should provide offsets into the original text! The Spacy tokenizer provides this, though the original wordpiece tokenizer provided for BERT does not. It is relatively easy to add offset information to that tokenizer, though. After you have start and end offsets for each token, it's not hard to just align tokens by offset overlap.
Re:How to calculate the alignment between BERT and spaCy tokens by PeterisP on Thursday July 15, 2021 @01:31PM (#27847337)
Huggingface tokenizers (e.g. BertTokenizerFast which can load the BERT model vocabularies) also can provide the offsets into the original text.