Companies are bad about doing this on purpose. If they set out to build AGI and accomplish something novel, just call that AI and go on fund raising from people who don't know better (or more likely don't care and just want to gamble with others' money).

Continuous RL in a sense. There maybe an undiscovered additional scaling law around models doing what you describe; continuous LLM-as-self-judge, if you will.

Provided it can be determined why a user ended the chat, which may turn out to be possible in some subset of conversations.

And also sometimes write down the conclusion and work backwards, without considering that the reason most likely for the conclusion isn't necessarily going to have the conclusion as the most likely conclusion — I hope I phrased that broken symmetry correctly.

I'm not following. Do you have an example?

The milliken oil drop experiment, “winning “ the space race, mostly anything C levels will tell the board and shareholders at a shareholder meeting, the American wars in Iraq and Afghanistan, most of what Sam Altman or Elon musk has to say, this list continues.

I think you're approaching it form very high level, when you should think about it from much lower level, i.e. success is being determined by stress/dopamine hormones or similar

The lower level seems to work eg, “Dopamine regulates decision thresholds in human reinforcement learning” https://www.nature.com/articles/s41467-023-41130-y

This article is kind of vague on that tbf: To conclude, we observed no credible evidence for a beneficial effect of L-dopa (vs. Haloperidol) on reinforcement learning in a reward context, as well as the proposed mechanistic account of an enhanced striatal prediction error response mediating this effect.

So your claim is that success stories are some kind of emergent phenomenon that comes from people chasing dopamine highs?

Is that controversial? I would say everything a human does is to feel better, and everything someone does that doesn’t make them feel better immediately is just done in the expectation of even greater pleasure later.

I don’t think the human experience can be reduced down to “if feels good is true then continue else try something else”. But I could be wrong.

Well mine can, with some tactics and strategy layered on top. If I do something I don’t like, I only do it because the payoff later makes it worth it (or at least I think it will from my current knowledge).

While humans seek “profit”.

It is important that “profit”, comes in various forms, which exchange rates are problematic to calculate (or maybe there can’t be any): not hungry, not thirsty, tastes good, not cold, feel safe, feel excited, feel righteous, feel powerful, listen to music, watch a movie, get curious, satisfy curiosity, laugh, love, sex, rock n roll.

I think it is both and this is very obvious if you read Pareto and Sapolsky.

It is more than just success too.

Most behavior we believe is some kind of rational action when it is really blind actions based on fiction or just completely random with rationalizations for the behavior after the fact.

someone once told me

> there are the facts we know and the stories we tell ourselves

and that really changed my outlook on life, the way i think about things, and basically everything else.

Aesop's fables, "sour grapes".

I did think of sour grapes (only thing that came to my mind) and was hoping for something better. Sour grapes doesn't seem too interesting. I think most people can tell you, if you actually ask them, what the differences between their actual successes and their wildest dreams are. But any improvement is still a success and I think that's valid.

This!

Yes, I've always thought that LLMs need the equivalent of a limbic system. This is how we solved this problem in organic computers. There is no static 'reward function'. Instead, we have a dynamic reward function computer. It decides from day to day and hour to hour what our basic objectives are. It also crucially handles emotional 'tagging' of memory. Memories that we store are proportionally more likely to be retrieved under similar emotional conditions. It helps to filter relevant memories, which is something LLMs definitely could use.

I think the equivalent of an LLM limbic system is more or less the missing piece for AGI. Now, how you'd go about making one of those I have no idea. How does one construct an emotional state space?