Re:SpaCy v3.0 Nightly
by ppod on Thursday October 15, 2020 @01:50PM
(#24791491)
I love Spacy, but I was recently trying to embed documents with it and it seems that it uses word-vector averaging for that. Is it possible to do sentence or paragraph embedding like sentenceBert?
[ Parent
| on HN ]
Yes, that's the sort of thing that's now much better. See here for discussion about using transformers in v3: https://nightly.spacy.io/usage/embeddings-transformers
Document similarity is a tricky thing to get into the main API though, which is organised around the idea of sending a Doc through a sequence of steps, each of which adds or updates annotations. The signature for document similarity is fundamentally different. In the end we decided to not try to shoe-horn it in. Not everything needs to be one function call. So document relation predictions (including document similarity) wouldn't be a standard pipeline component.