Scaling Transformer to 1M tokens and beyond with RMT

Re:Scaling Transformer to 1M tokens and beyond with RMT by kmac_ on Monday April 24, 2023 @04:37AM (#35685058)

I verified that it has all required facts (line of succession, current circumstances). I managed to get the right answer when got everything in context, but it failed again when all three abdicate (same context). Prince Harry was indicated once.

I tested GPT a lot in other domains, what I found that as long the information explicitly exists (connection between facts) then the responses are fine. I assume that if GPT will reach the state that it can infer new facts, we will be flooded with discoveries that require cross domain knowledge. Nothing like that happened yet.

[ Parent | on HN ]

Re:Scaling Transformer to 1M tokens and beyond with RMT by nullsense on Monday April 24, 2023 @05:57AM (#35685553)

>Nothing like that happened yet.

Feels like we're only one paper away now that the context window has absolutely ballooned.

[ Parent | on HN ]

Re:Scaling Transformer to 1M tokens and beyond with RMT by geraneum on Monday April 24, 2023 @10:40AM (#35688220)

This reminded me of "Two Minute Papers" YouTube channel where in most of the videos he always, "Two papers down the line and...". I think ML/AI is the main topic of his videos. Interesting stuff.

[ Parent | on HN ]

Re:Scaling Transformer to 1M tokens and beyond with RMT by nullsense on Monday April 24, 2023 @04:53PM (#35693122)

You just gave me a great weekend project idea. I need to clone his voice and whip up an interferface where you give it a paper and it summarizes it in his voice.

[ Parent | on HN ]