Some recent discoveries indicate that LLMs might remember information after seeing it only once. See e.g.: https://www.fast.ai/posts/2023-09-04-learning-jumps/