Would it make sense to rather compare distances to sub words rather instead full words ? I don't know if this would be possible because now you have to deal with sequence modeling to transform audio to sub words to words to sentences.