2015 called, they want you back! Now seriously, "just" does an amazing amount of work for you. How do you "just" make logistic regression write articles on politics, convert queries in SQL statements? or draw a daikon radish in a tutu?
Humans are "just" chemistry and electricity, and the whole universe just a few types of forces and particles. But that doesn't explain our complexity at all.
In other words, we too can't do three digit multiplication in our heads reliably, but can do it much better on paper, step by step. The problem you were mentioning is caused by the bad approach - LMs need intermediate reasoning steps to get from problem to solution, like us. We just need to ask them to produce the whole reasoning chain.
- Chain of Thought Prompting Elicits Reasoning in Large Language Models https://arxiv.org/abs/2201.11903
- Deep Learning for Symbolic Mathematics https://arxiv.org/abs/1912.01412
Whatever makes you think it’s necessary for AGI, when we don’t have it?
Even if it had 80% accuracy (optimistic) it would still he too mediocre to be used at any serious scale.
Most of the stupid crap that people give about degenerate cases where deep learning doesn't work (cartpoll in reinforcement learning, sine/infinite unbounded functions) are showcasing how bad gradient based training is - not how bad deep learning is at solving these problems. I can within seconds solve cartpoll with neural networks using neuroevolution of weights....
Because, with RELU activation, I’m fairly confident that the latter, at least, is possible.
(Where inputs are given using digits (where each digit could be represented with one floating point input), and the output is also represented with digits)
Like, you can implement a lookup table with neural net architecture. That’s not an issue.
And composing a lookup table with itself a number of times lets one do addition, etc.
... ok, I suppose for multiplication you would have to like, use more working space than what would effectively be a convolution, and one might complain that this extra structure of the network is “what is really doing the work”, but, I don’t think it is more complicated than the existing NN architectures?
If the weights are designed, and the network architecture allows something to hold the information needed, then there is really no obstacle to having it get multiplication entirely (not just 90%).
Now, would that be learnable? I’m not so sure, at least with the architecture one would use if designing the weights.
But,
I see no reason a transformer model couldn’t be trained on multiplication-with-work-shown and produce text fitting all of those patterns, and successfully perform multiplication for many digits that way.
And, by “showing all work” I don’t necessarily mean “in a way a person would typically show their work”, but in a easier-for-machine way.
Neither can people, for the most part.
I have more expertise in deep learning than anyone else here and the delusions of the incoming transformer winter will be painful to watch. In the meantime, enjoy your echo chamber.
I... I guess it's possible?
No, you don't. Looking at your experience, there is simply no way that you are the foremost expert in DL on HN.
What I know of your experience shows a low number of years of experience, a lack of papers, and a lack of true hands-on experience at the small number of companies in the world that have the resources to truly investigate large models. How can you know so much about LLMs without ever having the resource to train one?
I'm obviously not going to dox you, so you can easily just dismiss what I'm saying. But even just reading through your HN comments shows arrogance in your own knowledge (across multiple domains).
A specifically memorable quote is:
> I frequently create unique on the internet [words]
This is very true. Your erudition is apparently only matched by the uniqueness of the words you use when on the internet.
Meaning?