News for hackers. Stuff that matters.

A comment on Large language models are having their Stable Diffusion moment. Parent comment

Re:Large language models are having their Stable Diffusion m... by fock on Sunday March 12, 2023 @03:37AM (#35117738)
There's no memory or iteration log. You are just priming the (more or less deterministic even...) predictor with some history. You are doing a lot of work yourself, setting a factually correct context (and this has severe limits...).

As for "we just need a way to include iterations": isn't this what attention is supposed to do (kind of dynamically updating weights). The usual way to really update weights are variations on gradient descent. Can you link a paper outlining how we integrate your proposal into the current framework (which took around 20years to mature)? Otherwise your statements are Sci-Fi (at the flying cars-level).

[ Parent | on HN ]
Re:Large language models are having their Stable Diffusion m... by BulgarianIdiot on Sunday March 12, 2023 @08:31AM (#35119315)
Oh so there's no memory, just recallable history.

Oh so it's not learning by example, it's just primed by a sample.

Oh so there's no iteration of thought, just a loop of prediction.

OK.

Also, the papers you asked about: https://news.ycombinator.com/item?id=35115563

BTW, people are already implementing recursive/iterative queries on the ChatGPT API, and getting promising results.

[ Parent | on HN ]
Re:Large language models are having their Stable Diffusion m... by LawTalkingGuy on Sunday March 12, 2023 @12:41PM (#35122364)
The langchain project is an example of the iterative queries approach. It comes with constructs for working memory, factual lookup / calculation agents, etc.

https://github.com/hwchase17/langchain

The general (non-technical) guideline is that the LLMs can "answer" anything you just gave them the answer for. So you give it a problem, ask it how to solve it, tell it to use that method and explain the data it needs, give it that data, and then show it everything at once: "With this data you requested and summarized, use this technique to answer this question".

[ Parent | on HN ]