GPT-4o's Memory Breakthrough (Needle in a Needlestack)

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by robbiep on Tuesday May 14, 2024 @08:58PM (#40361917)

I’m not involved in the space, but it seems to me that having a model, in particular a massive model, exposed to a corpus of text like a book in the training data would have very minimal impact. I’m aware that people have been able to return data ‘out of the shadows’ pf the training data but to my mind a model being mildly influenced by the weights between different words in this text hardly constitute hard recall, if anything it now ‘knows’ a little of the linguistic style of the authour.

How far off am I?

[ Parent | on HN ]

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by int_19h on Tuesday May 14, 2024 @10:16PM (#40362405)

It depends on how many times it had seen that text during training. For example, GPT-4 can reproduce ayats from the Quran word for word in both Arabic and English. It can also reproduce the Navy SEAL copypasta complete with all the typos.

[ Parent | on HN ]

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by kaibee on Tuesday May 14, 2024 @11:06PM (#40362669)

Poe's "The Raven" also.

[ Parent | on HN ]

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by 19h on Tuesday May 14, 2024 @10:19PM (#40362421)

Brothers in username.. :-)

[ Parent | on HN ]

Re:GPT-4o's Memory Breakthrough (Needle in a Needlestack) by Salgat on Tuesday May 14, 2024 @11:35PM (#40362821)

Remember, it's also trained on countless internet discussions and papers on the book.

[ Parent | on HN ]