> Relevant past memories: > <splice in the N past messages with the most similar embedding to the most recent messages>

This is a really good idea. Presumably you'd keep the memory per-person.