The boring answer is to bash domain expertise and evals together until your data pipeline seems to be spitting out embeddings that give you acceptable results for whatever you’re trying to do.

I do wonder if there is some additional cleverness that’s possible with fast, long-context LLMs, maybe a summarizing/preprocessing step that can slice up the document into a sensible hierarchy of topics that can be used as the embeddings source. Never seen this in practice though.