Basically, if the knowledge exists online in a way that can be pieced together in a straight forward manner, GPT will figure it out, but for information that requires experimentation and creating new information to derive results, it won't be of much use. For example, GPT can't iteratively try different programming techniques to speed up a block of parallelizable code, it'll simply give you the best guess that it can find off google.
It won't iterate on its own, but you can do it. You can ask it for a list of things to try, and they will be different alternatives. You can also tell it the result of an experiment and it will often figure out what to fix.
If you follow the link I shared, some researchers automated asking GPT4 to write tests, running the tests in a sandbox, and feeding the results back in.
Re:From Deep to Long Learning
by machiaweliczny on Tuesday April 11, 2023 @12:33PM
(#35527513)
Why you assume it won't?
With access to python interpreter I don't see any issue
For example, I expect skill at writing some kinds of code to improve dramatically because running tests in a sandbox looks easy. It’s already being researched. [1] Extending that to device drivers might be a bit harder. Fuzzing is already mostly automated and smarter fuzzing could get pretty scary.
[1] https://nanothoughts.substack.com/p/reflecting-on-reflexion