Program Synthesis with Large Language Models

Re:Program Synthesis with Large Language Models by mountainriver on Tuesday August 17, 2021 @11:33PM (#28217496)

> We find that even our best models are generally unable to predict the output of a program given a specific input.

I’ve been thinking on this quite a bit, it’s an interesting property of language models in that they are essentially associative lookup machines.

It feels on some level that we need to be combining this kind of reasoning with something like world models.

[ Parent | on HN ]

Re:Program Synthesis with Large Language Models by tasdfqwer0897 on Tuesday August 17, 2021 @11:52PM (#28217612)

I personally agree that this experiment is evidence that there are certain problems that cannot be solved simply by making the models bigger, and one of the main research questions I'm interested in is what we need to do to elicit more reasoning-like capabilities from them.

There are people who fall more on the side of bitter-lesson/scaling-law-maximalism, and I think it's probably healthy and valuable that there are people in the research community placing both types of bet.

[ Parent | on HN ]

Re:Program Synthesis with Large Language Models by algo_trader on Wednesday August 18, 2021 @10:57AM (#28221910)

> elicit more reasoning-like capabilities from them.

But will a branch-and-search with some-noisy-evaluator get us there? Obviously this is easier with some domains.

Or maybe simply using increasing large "prompts" or "input specification" to specify the desired end result. There might be a while scaling law hiding there ..

[ Parent | on HN ]