Re:How to calculate the alignment between BERT and spaCy tokens
by cochne on Monday July 12, 2021 @10:37PM
(#27817156)
This is exactly why tokenizers should provide offsets into the original text! The Spacy tokenizer provides this, though the original wordpiece tokenizer provided for BERT does not. It is relatively easy to add offset information to that tokenizer, though. After you have start and end offsets for each token, it's not hard to just align tokens by offset overlap.
[ Parent
| on HN ]