For the best performance you'd probably want to use something other than CLIP's text encoder, but yeah you already have the text and could index it.

If enough people asked for it, I would consider adding a different embeddings models.