News for hackers. Stuff that matters.
Re:Building an early warning system for LLM-aided biological... by apsec112 on Wednesday January 31, 2024 @02:09PM (#39207966)
Way too many people extrapolate from "AI Foo can't do task X" to "AIs in general can never do task X", whether X is a good thing (like playing Go) or a bad thing (like helping build bioweapons). AI right now is pretty much the fastest-moving field in human history. GPT-4 can't revolutionize the economy, and it can't commit mass murder, but we simply don't know if GPT-7 can do either of those things (which likely won't be a pure LLM, but an integrated system with full audio/video abilities, robotics, long-term memory, etc.). We don't know that it can, but it also seems foolish to definitively say that it can't based on previous models, like the people who said that GPT-2's lack of logical reasoning ability proved that LLMs could never reason. We simply don't know, and we won't until it's actually built.
Re:Building an early warning system for LLM-aided biological... by solarpunk on Wednesday January 31, 2024 @02:12PM (#39208003)
we better make sure microsoft devotes all funding earmarked for openai to ensuring future versions of their product are incapable of creating bad outcomes.
Re:Building an early warning system for LLM-aided biological... by Terr_ on Wednesday January 31, 2024 @10:16PM (#39212396)
Meh, the humans at Microsoft already create bad outcomes on purpose--for consumers anyway.
Re:Building an early warning system for LLM-aided biological... by zeofig on Wednesday January 31, 2024 @05:04PM (#39210049)
On the contrary... way too many people see "AI can do X", "AI can do Y", "I can draw an imaginary line between X, Y and Z", "therefore AI can totally maybe do Z". The fact that we "just don't know" doesn't mean anything. You just don't know if Jesus will return tomorrow and bring the rapture.
Re:Building an early warning system for LLM-aided biological... by pixl97 on Wednesday January 31, 2024 @05:13PM (#39210154)
The probability that Jesus will return tomorrow is extremely low if not zero.

When talking about the probability of AI doing X or Y the probability is much closer to 1 if physics allows it.

Re:Building an early warning system for LLM-aided biological... by groggo on Wednesday January 31, 2024 @05:15PM (#39210185)
I appreciate that you acknowledge there's at least a chance.
Re:Building an early warning system for LLM-aided biological... by zeofig on Wednesday January 31, 2024 @05:35PM (#39210389)
If physics allows it... a considerable assumption. In my view physics (or mathematics, or whatever) doesn't even allow an LLM to "reason", let alone manufacture novel bioweapons. The probability of Jesus returning is considerable too if we assume the bible is true.
Re:Building an early warning system for LLM-aided biological... by marcellus23 on Wednesday January 31, 2024 @06:45PM (#39211131)
What if you ignore LLMs specifically? I think that's the point that the GP was making. Do you believe it's mathematically impossible for any artificial machine to "reason"? Or just LLMs?
Re:Building an early warning system for LLM-aided biological... by drowsspa on Wednesday January 31, 2024 @07:14PM (#39211347)
I'm guessing the answer will be soul or something of sorts.

Not even based on the previous comments, even non-religious people seem to have this supernatural view of a soul.

Re:Building an early warning system for LLM-aided biological... by Libcat99 on Wednesday January 31, 2024 @09:44PM (#39212248)
If humans can reason, then physics allows it, and software can too, eventually.
Re:Building an early warning system for LLM-aided biological... by knightoffaith on Wednesday January 31, 2024 @07:28PM (#39211461)
Why doesn't physics or mathematics allow LLMs to reason?
Re:Building an early warning system for LLM-aided biological... by Vecr on Wednesday January 31, 2024 @10:40PM (#39212514)
https://news.ycombinator.com/item?id=39173034 If this paper replicates, LLM-like large transformer models can approximate Solomonoff induction pretty well, with other neural net types showing clear evidence of generalizing outside the training distribution. If substantial efficiency improvements get implemented and new architectures get developed, that's game over. The AI is smarter than you, and can trivially be made into an agent by having the approximate Solomonoff induction section simulate the behavior of an agent in the real world, and then having another section change the real world to match the simulation by controlling the real-world agent, creating a full cybernetic system, a closed loop.
Re:Building an early warning system for LLM-aided biological... by croniev on Wednesday January 31, 2024 @07:15PM (#39211353)
Would you say that GPT-4 can reason now? I am not convinced this is case, it seems like it has just become more consistent at providing us with an output that we consider reasonable because it was engineered precisely to do that.
Re:Building an early warning system for LLM-aided biological... by fragmede on Wednesday January 31, 2024 @07:58PM (#39211690)
then you have to define reason, and it gets all philosophical. suffice to say, it's able to take a implies b and b implies c to get a implies c, and make up things along the way so calling it glorified autocomplete is a dishonest representation of its abilities. it doesn't love or feel jealous but it's very good at writing essays about whatever I ask it to. it doesn't need to do more than that to be useful to me, today.

the human words we have for consciousness aren't good enough to describe what ChatGPT does. thinking, reasoning, understanding. it processes. it computes. it matrix multiplies. it takes the dot product and the determinant. there are eigenvectors and eigenvalues. it's tensoring and outputting the code and prose I asked it for.

Re:Building an early warning system for LLM-aided biological... by Nifty3929 on Wednesday January 31, 2024 @07:31PM (#39211485)
It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything.

They literally just make stuff up (technically just a continuation of whatever you fed in), which usually sounds good, is often true and sometimes even helpful. Because those are qualities of the training data that was used and is the basis for the stuff it’s making up.

Re:Building an early warning system for LLM-aided biological... by foo3a9c4 on Wednesday January 31, 2024 @09:01PM (#39212026)
> It’s definitely not the case. LLMs of any sort do not in any sense reason or understand anything.

This seems like a claim about the way that the LLM neural net algorithm works. But AFAIK no one has a good understanding of how the LLM NNs work.

Why are you so certain that the LLM NN isn't doing the reasoning-algorithm or the understanding-algorithm?

Re:Building an early warning system for LLM-aided biological... by habinero on Wednesday January 31, 2024 @10:04PM (#39212345)
Neural networks are not new, and they're just mathematical systems.

LLMs don't think. At all. They're basically glorified autocorrect. What they're good for is generating a lot of natural-sounding text that fools people into thinking there's more going on than there really is.

Re:Building an early warning system for LLM-aided biological... by foo3a9c4 on Wednesday January 31, 2024 @10:30PM (#39212470)
> Neural networks are not new

I agree. The McCullough-Pitts paper was published in 1943.

> they're just mathematical systems.

What do you mean by "mathematical system"? AFAIK the GPT4 model is literally a computer program.

> LLMs don't think. At all.

This is the same assertion that OP made and I'm still confused as to how anyone could be certain of its truth given that no one actually knows what is going on inside of the GPT4 program.

> They're basically glorified autocorrect. What they're good for is generating a lot of natural-sounding text that fools people into thinking there's more going on than there really is.

Is that an argument for the claim "LLMs don't think."? It doesn't seem like it to me, but maybe I'm mistaken.

Re:Building an early warning system for LLM-aided biological... by Al-Khwarizmi on Thursday February 01, 2024 @03:21AM (#39213811)
Not new, but we don't understand how they work at the large scale.

I don't think reductionistic arguments hold much water. Sure, neural networks are just matrix multiplication. In the same way that a brain is just a bunch of cells. Understanding the basic building blocks doesn't mean understanding the whole.

We can always say that LLMs don't think if we define "think" as using a biological brain, but the fact is that they generate outputs that from the human perspective, can only plausibly be generated via reasoning. So they, at the very least, have processes that can functionally achieve the same goal as reasoning. The "stochastic parrot" metaphor, while apt in its day, has proven obsolete with pretty much all the examples of things that LLMs "could not do" in early papers being actually doable with the likes of GPT-4; so arguments against the possibility of LLMs reasoning look like constant moving of the goalposts.

Re:Building an early warning system for LLM-aided biological... by DougBTX on Thursday February 01, 2024 @04:13AM (#39214061)
> and they're just mathematical systems

Obvious question: can Prolog do reasoning?

If your definition of reasoning excludes Prolog, then... I'm not sure what to say!

Re:Building an early warning system for LLM-aided biological... by laughingman2 on Wednesday January 31, 2024 @09:07PM (#39212061)
Research done by Prof. Subbarao on reasoning & planning with LLM seems to point in the negative.

https://www.youtube.com/watch?v=uTXXYi75QCU

He has lots of good threads distilling his research. https://twitter.com/rao2z

Re:Building an early warning system for LLM-aided biological... by visarga on Thursday February 01, 2024 @01:37AM (#39213291)
> Would you say that GPT-4 can reason now?

Let's assume reasoning entails going beyond the stochastic parrot level. Can LLMs have skills not demonstrated in the training set?

Here is a paper demonstrating that GPT-4 can combine up to 5 skills from a set of 100, effectively covering 100^5 tuples of skills, while only seeing much fewer combinations in training on a specific topic.

> simple probability calculations indicate that GPT-4's reasonable performance on k=5 is suggestive of going beyond "stochastic parrot" behavior (Bender et al., 2021), i.e., it combines skills in ways that it had not seen during training https://arxiv.org/abs/2310.17567

So they show ability to freely combine skills, and the limit of k=5 measured in this benchmark illustrate that models do generalize. They are able to apply skills in new combinations correctly, but there is also a limit.

The interesting part is how they demonstrate that, let's say on a topic with n=1000 samples in the training set it is impossible to have sufficient training examples covering tuples of 5 skills, but models (mostly GPT-4) can handle it. Other models top out at tuples of only 2 or 3 skills.

Models combining skills in new ways are not just parroting. They can perform meaningful work outside their training distribution.

Re:Building an early warning system for LLM-aided biological... by pushfoo on Thursday February 01, 2024 @03:35AM (#39213886)
TL;DR: They're close enough to make people argue and publish papers about similarities to the human hippocampus

I have a hunch these models are approximating an important subset of what we call reasoning. In dangerously reductive terms, it's a question of how closely and how much of a function's output we can approximate.

There was at least one paper[1] showing similarities between AI models and the hippocampus. That lines up with another part of human neuroscience: at least part of human reasoning appears to take place inside the hippocampus itself [2].

From my neuroscience background, the takeaways seem to be:

* Carmack is right: we're missing some important bridging concepts for AGI.

* Whether current LLMs can reason depends on how you define reasoning

I'm unsure whether finding answers in those areas would be good thing. Instead of alignment issues or misuse, I'm more worried about how quickly people would overreact to it. We might already be seeing that in business.

1. https://arxiv.org/abs/2103.07356

2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3312239