You seem to be talking about neural Turing machines: https://arxiv.org/abs/1410.5401

Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.