In other words, I don't care why people do things, only that they do.
I’m not aware of many examples of starry-eyed divas achieving great results. Usually you hear about them but only because they are exceptional cases, not the norm. It’s a matter of practicality and not virtue (to say otherwise is purely a straw man argument).
This is really overconfident.
That publicity isn't causally connected to success is belied by the existence of the advertising industry. While generally refuting across industries, it is worth noting that the most dominant AI company - Google - happens to be in this industry. They are explicitly known - having publicity for - their generous compensation packages. This is because of a causal model of talent attraction.
Success is obviously causally connected to publicity and the idea that it isn't isn't well supported by the evidence. Contrary to your assertion, it was not a safe assumption. Your appeal to Satoshi is an appeal to authority, not a causal model of its shielding off from project impacts.
The argument was about publicity as a reward motivator, not publicity itself, as a causal relation to success.
To phrase it plainly: Which first-time founders do you think Paul Graham or Keith Rabois would more likely fund: Those who aspire to solve a problem with the world that they care passionately about? Or those looking for money or fame? Last time I checked the latter case would be seen as a strong negative. The appeal to authority argument doesn’t apply in this situation because the VC portfolio performance is causally related to how accurately they predict future success of a company.
On the scale of a smaller project like this, a common failure mode is for a maintainer to stop caring about the project and go to the next thing that motivates them. Someone else may attempt to use the code or project without understanding the theory behind it. And even worse: every time this happens is a signal that this is acceptable.
AI is a different beast. Software bugs with big AI systems will become more costly, and eventually deadly. Unfortunately I’m not sure what can be done about it without a global totalitarian regime to ban its use entirely (which is not an idea I support anyway). Eventually the broken clock will be right and some profit-driven AI project will succeed in making the world a not better place, if we are even around to notice :).
I would advise deeper thought into these topics when convenient. Read Nick Bostrom’s Superintelligence book or watch his talks, at least one of which was at Google HQ.
I think someone should train ChatGPT or similar to argue or teach traditional AGI Philosophy/Ethics and hopefully that will move the needle somewhat more than the OpenAI nannyism we have now.
You are again making unsafe assumptions.
I've read Superintelligence. I've read the Sequences. I've even read the part of the sequences where Yudowsky openly admits that he believes Peter Norvig to be above his level - and it is worth noting that Norvig disagreed with Yudowsky on the topic of AI risks.
I've gone much farther too. I've read PAIP and AIMA, both written by Norvig. I've also read Deep Learning which is by Goodfellow et. all. I've read some of Reinforcement Learning by Sutton and completed successful reinforcement learning projects. Hell, I've even read the initial book on Game Theory, though I've been more impress by the papers advancing the state of the art in that field. I know things about the subject that aren't published in any textbook and I know some things which, because I'm so near the frontier, I suspect I'm among the only people who know. In the distant past, I read books on genetic and evolutionary algorithms. I've implemented some. A copy of Science magazine sits on the desk beside me, because there is material in it that I wished to study so I bought it. Beside that is a paper on bayesian modeling in imperfect information games. I've taken courses in Control theory. During my employment at a what used to be a FANG, but is now MANGA, I worked on machine learning systems.
You are making unsafe assumptions about me.
> The argument was about publicity as a reward motivator, not publicity itself, as a causal relation to success.
Reinforcement learning in the context of ensemble models refutes this argument. It is possible to compose agents out of agents with different reward signals. The mathematics of ensemble learning tell us that when models are good for different reasons, the model built out of the ensemble of those models can be better than any model was individually.
> Which first-time founders do you think Paul Graham or Keith Rabois would more likely fund: Those who aspire to solve a problem with the world that they care passionately about?
It is worth reflecting on the fact that the founder of OpenAI has had the strongest possible endorsement from Paul Graham. He was claimed to be among the greats before his successes: Paul Graham put him among Steve Jobs and Elon Musk. When Paul Graham stepped down from YCombinator, he was so convinced of Sam's skills that he put Sam in his place. Later Sam started OpenAI.
You are giving an argument here for your initial hope that companies like OpenAI wouldn't be fame seeking. Is this your intention? I get the impression from things like "nannyism" that you are actually intending to argue the opposite.
> Eventually the broken clock will be right and some profit-driven AI project will succeed in making the world a not better place, if we are even around to notice :).
One thing that Yudowsky and people highly informed by him miss is that his teaching about rationality in terms of overcoming bias isn't actually supported by what biases really are. It really helps to use simpler models of cognition. For example, in a chess playing AI, you end up seeing a dual mode process just like you do in the brain. Bostrom and Yudowsky would both have you believe that there is a better agent so much so that it puts all existing agents to shame. The thing is, we actually do know how to calculate the perfect agent through working backward from the outcomes through the transitional structure of the game to its start. The issue is that the number of states which are possible in the game is very high - comparable to the number of atoms in the universe high. This is sometimes called the combinatiorial explosion.
It ends up that isn't physically realistically even in simple problems to have a perfect intelligence. So what the brain does and what chess engines do is that they have a dual mode system. The first part solves the general problem, but it has error. Then the other part solves for a specific situation: because the specific situation is less general than the general problem the error-laden heuristic it provides always has error and is actually algorithmically intended to be used to improve upon the approximation. Yet you can't induct on the omnipresent ability to make progress in the specific situations to the ability to make progress in the general situation. Yudowsky and Bostrom and many people actually do try to do this sort of inference based on observed ability to make progress, so they have extremely wrong intuitions about what it is possible.
Generally speaking, everyone who speaks about cognitive biases pejoratively tends to suffer from this error. A good example is that the author of the OvercomingBias blog has actually articulated bias toward things that benefit a person as an example of a bias. Notice, in the dual mode systems we are talking about, this isn't a bias. Mathematically the assumption that people do this and we should do this corresponds with equillibrium approximation improvement. You see it show up in things like the minimax algorithm, not because it is irrational, but because the minimax algorithm corresponds with nash equilibrium approximation error reduction. The causal explanation of its purpose doesn't correspond with bias, but because these people aren't working from models of cognition, they just observe error and think - ah, bias, lets overcome it. In actuality almost all the seeming bias in humans are better understood in terms of performance optimizations that are necessitated by the problems being solved being very hard.
Anyway, I'm really just trying to argue you were being overconfident. I don't think your belief that publicity
See what I mean about failing to truly grapple with intractability? He doesn't address the subject as seriously as it ought to be! It ought to be taken with deadly seriousness, because it isn't a niche thing irrelevant to the discussion.
Notice Turing's halting problem applies when discussing agents reacting to other agents! Notice it!
Notice that decisions in game theory, especially the equilibrium policies we can make claims about, are defined with respect to other policies. Notice it!
Notice that the general category of decision problems we are discussing with regard to intelligence is actually of the sort in which we are dealing with Turing's halting problem and Gödel's incompleteness. This is actually not an irrelevant category which he is excluding - he just accidentally skipped past, with a cheery "most are like this" handwave - most questions actually relevant to intelligent agents.
It wasn't him being clever. It wasn't him being wise. It wasn't him being formidable. It was him clinging to the lights, because he wants to think within the lights.
In contrast, Norvig just admits we have to give up on certain aspects of correctness due to intractability. It is so much more formidable, because he doesn't try to operate just under the light, but explicitly calls attention to the darkness and talks about the implications of grappling with it.
I would say that Bostrom does the same thing, but he gave more attention to the idea of going faster than humans can and less attention to the idea that there is a physical limit to the speed of light. He focuses more on the physical possibility of improvements due to the benefit of increased traction and less on the inevitably of encountering the impossibility of traction.
[1]: https://en.wikipedia.org/wiki/Curse_of_dimensionality
[2]: https://en.wikipedia.org/wiki/Combinatorial_explosion
OpenAI has a pretty good introduction to the Bellman equations in their Spinning Up in RL lessons [3]. Sutton's work in Reinforcement Learning also talks about Bellman's work quite a bit. Though Bellman was actually studying what he called dynamic programming problems his work is now considered foundational in reinforcement learning.
[3]: https://spinningup.openai.com/en/latest/
Uh, and for the dual mode observations the person that brought that to my attention was Noam Brown, not Bellman or Norvig. If you haven't already checked out his work, I recommend it above both Norvig and Bellman. He has some great talks on Youtube and I consider it a shame they aren't more widely viewed [4].
That the causal model supports publicity seeking leads us to ensemble models. When models are good for different reason, the ensemble of the models ends up better than any individual model. Reinforcement learning research has shown you can successfully train an agent out of decomposed reward signals by building an ensemble model atop them.
The fact that the causality says publicity matters means that agents which recognize the importance that publicity contribute to the solution actually do have the expectation of being part of the solution.
It is very common to see this talked about in terms of diversity improving solution quality when talking about it in the context of companies and it is generally considered a good idea to have a diverse team as a consequence.
Anyway, I'm mostly responding because I disagree with apriori declaration that all who disagree are attacking a straw man.
I think that was overconfident, because the causal structure of publicity and its relation to outcomes disagrees with that.