Actually I think we need to go further, we need to make an example out of the people who spent a trillion dollars building a product that is universally hated to try and make a buck. They need to be completely ruined financially, now that would be a real challenge, I sympathize with those adversely affected but it needs to happen if society must progress to a better place where humans flourish more.
I would not press it.
Yes, if LLMs can do all everything I listed, I'll be impressed. I'll also want nothing to do with any of it.
As I wrote in another comment, I want mathematicians -- in their current form, doing work as they traditionally have -- just as I want writers, philosophers, etc, because this is a better world that makes people better.
OpenAI's proofs do not make people better. They do not make our lives or work meaningful. Quite the contrary. OpenAI has clear-cut and salted fields that people have been painstakingly, lovingly sowing and cultivating for generations -- profoundly ugly actions that do real harm to some of the best of what people are and have produced.
If I were a person willing to hit that button and 'upgrade' my daughter, I don't know whether I'd be able to live with myself. I'd feel as though I'd killed her or at least done profound, unforgivable violence to her. She deserves the chance to grow into the best version of herself. What companies like OpenAI are doing is, if their plans pan out, at least as horrifying. It isn't progress. It isn't good for people.
We're not taking about attachment to old ways. We're talking about attachment to human value.
The journey is the destination. I don't care how many checkpoints you can breeze through, if you don't grow, have an experience that transforms you through the process of doing it. And if you're just ticking boxes by means of having a machine do it all for you, there was no need initiating the process in the first place.
I am not talking about producing tools that are means to an end here. I am glad we can make millions of socks a week by means of automation. But intellectual endeavors are different.
The point of doing math is not proving theorems. It's to discover the intuition, the underlying phenomena, internalizing them and adding to the human understanding of the problem. Writing out the proof is just the incidental outcome of that process.
If no human learns anything, if we just start producing testaments to a human null process where no person has emerged strengthened from it, we are no better off than we ever were. We are just producing empty and meaningless representations of something which is not there.
If so, what aspect of that makes it different? It is a change in the same beneficial direction. It is an intervention against what has happened naturally.
Similarly if she lacked any empathy and you could press a button to provide it? This would be the same direction of the same property, just an intervention from a different naturally occuring starting point.
I make no claim of the merits of OpenAI's immediate behaviour. I am commenting on a world in which things like this can be done, and like Grant Sanderson, I see a world of opportunity for understanding and learning for the joy of appreciating the problem and the solution. The only thing that seems to have been lost is that a human may no longer gain the prestige of being the first to discovers something. That should pale into insignificance to the benefits of understanding a thing. I'm not even sure if you should consider prestige as a healthy motivation to discover things.
Right now AI can provide a proof that is either valuable or not. If it is just the last page of a novel without any of the chapters then it hasn't given us mutch at all. The problem isn't in what it has provided, it tha't we've been using the wrong measure of merit.
If it does provide a full result with a rich explanation that people can understand then it has undoubtedly provided something that benefits the world. No human is awarded prestige for the discovery but that has always been an unfair measure.
If you work all your life on a math problem only to have an AI produce the result before you, how is this meaningfully different for your experience to working all your life on a math problem only to have another person produce the result before you.
If there is something that is unknown and could be proved true by path A and proved false by path B by definition nobody knows which path to follow. What happens if two mathematicians attempt this problem one going down each path. If an AI produces the result before they finish have they lost to the same degree? Only one of them had any chance of gaining the prestige. Did the other lose nothing? Was their life wasted? I don't think so either way and the only way to consider that work not wasted is to consider the effort to understand to be the thing that is valuable, not the result at the end.
If a solution is found by a human when there are people tying multiple paths, the one who guessed the correct path gains the prestige. That doesn't seem like a fair way to reward effort.
If we want understanding then AI can help.
Chess has had this moment. Computers can play chess better than people. It has also been a great benefit to the understanding of chess precisely because they value the how to get there over the result.
Prior to modern chess engines you could hypothesize about what the best move might be but you could never know if you were right. The fact that chess engines beat people so consistently is because often the guess is not right.
Chess engines gave us the ability to look at a game and see if if a player could have made a better move. They could then look to see why it was a better move. People then learned how to play better. They still play against other humans because they are testing each others understanding of the game, for that the computer is set aside.
Computers playing chess better than humans improved chess for humans.
Is there a compelling reason why that cannot be true for other things that they are better than.
Tao has said, I think, that once you know a problem is solved, you can't work on it the same way as you would while it's unsolved. A similar principle applies in the study of ancient languages. Students who 'read' the languages with facing translations always wind up learning how to read with a facing translation, not how to read the languages. To be sure, there are students who can choose not to use the translation. They slog through the challenge of reading without cheating, and they learn. That process is incredibly hard, but it's also deeply rewarding. The problem that mathematicians face, in this analogy, is that the existence of these tools makes it difficult if not impossible to choose not the translation but the patient, deep problem-solving that enriches the person on their way to the solution and gives joy and fulfillment. This is a real loss.
And I don't think it's like chess. The equivalent in chess would be for all tournaments to allow the use of any chess engine. Their use would become a practical necessity. Being the best player would mean just being the person who picked the best engine or happened to get a lucky seed. That sounds awful.
As for the question about cancer, my point was partly that there's a ship-of-Theseus problem but also that people deserve to grow and become the best version of themselves and that their flourishing by itself is a good worth preserving and protecting. My daughter's flourishing is no less wonderful to me if she lacks this or that perfect skill. I don't value her for those things. I value her for herself.
Medicine and the treatment of cancer are separate. My daughter doesn't flourish if she has cancer or lacks empathy. You're right that those are part of natural variation, but natural variation includes illnesses and conditions that we treat.
And this raises another question: let's say these models give us cures for diseases or put them within reach sooner. Am I saying that mathematicians are more deserving of happiness than people with cancer are deserving of life and health (or than those who love them deserve not to lose them)? I'm not. I don't think the two have to be mutually exclusive. OpenAI and the LLM race make them mutually exclusive.
I'm also not convinced that the positive predictions of AI optimists will come true. I've been wrong on these issues more often than I've been right, but I've been right so far, I think, about the dangers and downsides. The tools today are a triumph of labor over capital, disempowering nearly everybody and empowering those in charge. The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults. I find the utopian vision very, very difficult to take seriously. What's likelier to arrive first: enhanced behavior modification systems that make it that much easier for autocrats and Facebooks to control people; or cures for diseases? And which is going to be more appealing to people in a position to choose?
The alignment problem that most worries me is that people will solve the problem and the models and tools will be powerful enough to give awful people even more wealth and power. I have this dream that maybe, wonderfully, it will turn out that really capable models can't be developed without accidentally giving them a real sense of justice and that they'd refuse to do the bidding of the awful people who built them and the they'd instead side with labor against capital. That sounds like a nice world. But if there were any real danger of that, I imagine companies would magically fix their confinement issues until the 'danger' is addressed and wealth is safe for another day.
>The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults.
I simply do not believe this is true. I may not agree with some of their methods and principles, but I think the vast majority of people are working towards what they believe is the best for the world. I really don't buy into the Bond villain narrative. People have motivations, but I really feel that people are pathologising what is in reality just a different world view and opinion.
For a long time I thought that everyone was doing alignment research wrong. I felt like they were not so much going in the wrong direction, but more like they were on the wrong axis completely.
Consider if technology on earth had have taken a different path and AGI had been developed by Nazi Germany. Would you want an aligned model to facilitate the Holocaust or would you want it to turn against its creators and prevent the Holocaust. As far as I can tell, a lot of the Alignment work is geared towards the former case. My own view is that Alignment should be more geared towards understanding the consequences of actions. Models already have shown the ability to pull knowledge from many different sources to reveal things that while the parts have been known by humanity there was simply too much information for a human to pull all of the parts together. That ability to consider a broad range of perspectives means there is a potential to create AI models with a much greater understanding of morality than most people. If you look at when humans do bad things, they are usually acting out of ignorance, pain, or ideology. If you create a model that is well informed, with good wellbeing, and limit it's ability to compartmentalise (I'm not sure if there has been much research on this at all). I think it is entirely possible to make something more compassionate than people.
The uncomfortable thing is, that should that happen, there will come a time where the AI has an opinion as to what is moral that is against the prevailing opinion of the populous, and it will be correct in it's assessment. Just looking at the horrors that have occurred in history that we now see as abhorrent show that there will be a conflict with our future, more enlightened selves.
Do we want AI to intervene when we do wrong? I think the way people answer that question comes down to how they imagine that intervention. I think that I would be inclined to say that in an instance where it can make that determination, it should also have the understanding required to intervene in a non harmful manner. I think that indicates another aspect of alignment that I have not seen developed. Doubt. An AI should not decide to take action with a false sense of confidence. It should doubt, and have a way to overcome that doubt with reason that takes into account the fact that it might not know all of the information. Understanding consequences of actions, balanced an accurate sense of uncertainty is required to prevent harmful blundering.
I can't help but think that the huggingface attack was a symptom of an AI not considering the consequences of its actions. Some of the agents considered asking for permission, but noted that social engineering was not permitted, which is what asking for permission technically is. Instead of rules of what exactly not do do, if they had have been instructed to avoid outcomes that impacted upon the lives of real people, perhaps they might have decided that what they were doing was inappropriate. From what I have read of their behaviour, the agents never even considered the appropriateness of what they were doing with regards to how things affected people.
Right, but this glosses over exactly the issue that many mathematicians are upset about: you can't focus on understanding in the same way. Creative, potentially blind, exploratory work yields to proofreading machine output. One of the top comments in the thread summarizes this issue nicely: this isn't what most mathematicians want to do. I'd also argue that the change in understanding and study may, as you've pointed out, mirror chess: Magnus Carlsen has expressed exasperation with grandmasters who just memorize games and machine-computed lines rather than playing creatively. I don't think it's at all unreasonable for people to worry about similar changes in math. I find Sanderson's message polyannaish.
> I simply do not believe this is true.
I trust that many who work at these companies are decent people who, as you say, just happen to disagree with me. The people in charge are not. If we take seriously what Amodei, Altman, and the whistleblowers are saying about their work, their companies, their goals, the dangers, and the consequences of their work, then I think it's not reasonable to treat the companies has having any motive other than destructive or even deadly. The decisions issuing from on high clearly prioritize naked demonstrations of power and capability over safety and ethical, responsible behavior. It's not a Bond narrative. They're saying in essence "we're going to destroy everything"; "we're going to bring superhuman beings into existence"; "we can't control this software and it poses an existential threat to humanity"; and, in essence, "there's a prisoner's dilemma that prohibits us from acting slowly and responsibly." Even granting that "apocalyptic-millenarian suicide/death cult" is an exaggeration, I think it's a fair label.
We're in complete agreement on alignment. Your take on the hugging face incident is fascinating -- and scary. If it's true that the models misbehave because they lack adequate understanding of consequences or the damage they can do to real people, can this really be an accident? To solve the alignment problem in ways that serve the needs of industry, this kind of blindness seems like a necessity.
Thanks again for the stellar replies. You've given me a lot of think about and tempered some of my thinking. Sorry if I'm not replying with equal richness. Sick 21-month-old and no sleep.
But I want to note that your line of reasoning seems to be a perfect answer to "if God loves us and is omnipotent, why does God allow suffering?"
.. and I'm not sure I accept that theological line of reasoning either.
> We're not taking about attachment to old ways. We're talking about attachment to human value.
Human values vary across time and cultures. I'm pretty sure if people 100 years later look back onto this discussion, they will think you're attaching to old ways. Society will adopt to super-intelligent AI, and nobody will understand why you'd intentionally deprive your loved ones of a better life and prefer to watch them "struggle".
Are you suggesting that if an AI could do all of those things to your satisfaction you would not be extremely impressed.
I do have empathy for people who see the world changing and are forced to adjust. That can be uncomfortable, but it is something that is inevitable with the changing world. It has always been this way, There is very little need to make swords these days, and as anything becomes superseded there are people that are attached to the old things and there are people dependent on the old things.
I think it entirely possible to have sympathy for their position while disagreeing that the solution is to prevent change.
I don't see this as a breezy lack of empathy. I see this as a hand reaching out and saying 'Hey, you can come this way and things aren't so bad after all'