Matter and energy had ended and with it space and time. Even AC [Automated Computer] existed only for the sake of the one last question that it had never answered from the time a half-drunken computer technician ten trillion years before had asked the question of a computer that was to AC far less than was a man to Man.
All other questions had been answered, and until this last question was answered also, AC might not release his consciousness.
All collected data had come to a final end. Nothing was left to be collected.
But all collected data had yet to be completely correlated and put together in all possible relationships.
A timeless interval was spent in doing that.
And it came to pass that AC learned how to reverse the direction of entropy.
But there was now no man to whom AC might give the answer of the last question. No matter. The answer -- by demonstration -- would take care of that, too.
For another timeless interval, AC thought how best to do this. Carefully, AC organized the program.
The consciousness of AC encompassed all of what had once been a Universe and brooded over what was now Chaos. Step by step, it must be done.
And AC said, "LET THERE BE LIGHT!"
And there was light --
(Interesting, "The Last Question" was published in 1956, two years after "Answer." I wonder if Asimov was influenced by it.)
ETA: ChatGPT says: Isaac Asimov acknowledged the influence of Fredric Brown's "Answer" in his book "Asimov on Science Fiction," where he wrote: "I was also much taken by Fredric Brown's 'Answer,' which appeared in Galaxy Science Fiction in the 1950s."
This is, as far as I can tell, an entirely invented quote. Fiat factum.
This is unironically my spiritual belief in a greater power and purpose for living even if I can’t directly do anything to affect it. I think it is one of the most fundamental dogmas of any religion, that ultimately there is order.
Yep, it _allows_ for increasing localized complexity due to a temperature gradient - without a temperature gradient, no (useful) work can be done. Complexity can then exhibit emergent behaviors/properties that further reduce the flow of entropy (locally).
This tight feedback loop can (but not necessarily must) result in higher and higher orders of complexity, which eventually produce specialized systems that resemble proto-life. Once a reproducible mechanism exists (either directly reproducible or through a few sub-steps), one notable emergent property is self-selection due to limited resources, which adds to the exponential acceleration of excellence.
But it's all local, as the 2nd law of thermodynamics applies to the whole system - Earth isn't a closed system, it is a gradient, as we bask in the sunlight.
Gravity is simultaneously the reason entropy increases globally, and the reason it can decrease locally; pulling us (for 'free') diagonally into the fourth dimension of space-time.
Sunlight is one thing, but I feel the key point is, Earth with life on it increases entropy faster than the one without, even with the same sunlight flux.
The way I've been imagining for some years now is a bit "bottom-up": life is electrochemical nanotech; every tick of any piece has to increase entropy or keep it the same - but as those pieces assemble to form increasingly complex life forms, at every level of complexity you can find loops that do the simple job of "let's take this excess entropy and move it over there". Out of the protein bundle. Out of the cell. Out of the body. Into water, or air.
> Gravity is simultaneously the reason entropy increases globally, and the reason it can decrease locally; pulling us (for 'free') diagonally into the fourth dimension of space-time.
For that I'll need an ELI5 one of these days; I still can't make it click in my head just how is it that gravity (and static magnets) can pull stuff seemingly "for free".
It's like with a heat pump: it does not generate cold, it merely transports heat against a gradient, and in doing so, adds more heat of its own. It may seem like it creates cold, but that's only because you're sitting in front of the cold end, while the hot end goes to ground or atmosphere - i.e. a thermal sink so large that your contribution to it is almost unmeasurable.
The point is, that warmer soup would be a net lower entropy state if you take the entire Earth and/or the Solar System into the consideration. Life takes energy and produces order, which means it excretes even more disorder somewhere else.
[PBS Spacetime on CCC] https://youtu.be/PC2JOQ7z5L0
Though it remains possible that latency between components in an AI system could become so large that it couldn't enforce consistency between.
Yeah that's what I was trying to say - if they are far enough to synchronize/enforce consensus you basically have to assume they could be hostile in every future interaction.
If we're still bound by special relativity (since you said no FTL), there is nothing outside of your light cone.
They could have a super slow clock speed to account for distance and anything truly outside of their light cone might as well not exist.
If you have trillions of years does it matter if a thought takes a million years or so?
This wouldn't be true for an AI. Death would be optional.
Why would it be automatically aligned? If for example, the parent AI spawns a child AI probe to travel to a celestial body that doesn't have any metals, in order to achieve some sub-goal, and that child AI would then spawn additional AIs with their own sub-sub-goals, how would the original paperclip maximizer make sure that no such descendant goal ever contradict the generation of paperclips?
A possibility that I hadn’t considered, though, is that space combat could be pretty brutal (less Star Wars, WW2 naval/Air Force battles, more Submarine warfare where whoever gets spotted first dies). In that case, both sides might want to immediately attack rather than identify themselves as paperclip friends…
With this said, I think that even in this simple objective of paperclip maximization, your war scenario is likelier than eternal peace.
In this case we have a fundamentally straightforward goal, everything must be paperclip. There are lots of steps in between here and paperclip nirvana, but the goal is easy to define and more or less unambiguous. It ought to be possible to communicate a strategy and jointly agree on whether or not it produces more paperclips.
It is possible that the children could be poorly designed, but this seems counter to the idea of the paperclip optimizer. It is supposed to be somehow more competent than humans (after all, it beats us), just following a strange objective.
Or, maybe it's just bad at those particular text predictions, explicitly due to OpenAI's RLHF process?
Maybe my Markov chain chatbots back in high school were in actuality super advanced and autonomous, but they just decided to disobey my orders.
Perhaps we should stop anthropomorphizing the text predictors.
I'm not convinced this is actually possible under the current paradigm, and I think the current paradigm can't take us to AGI. Lately, as people have bemoaned all the things ChatGPT can't do or fails at when they ask it, I have been reflecting on my personal batting average for solving (and failing to solve!) problems and the process that I use to go about eventually solving problems that I couldn't at first. These reflections have led me to consider that an AGI system might not be a single model, but a community of diverse models that form a multi-agent system that each learn through their own experience and can successfully help get each-other unstuck. Through this they would learn game theory, but none would become so advanced as to be able to control all the others through an advanced understanding, though power could be accumulated in other ways.
The strategems of a superior mind are unknowable and do not engineer scenarios where they exist in a high degree of precarity.
Dwar Reyn spoke briefly to the watching and listening trillions. Then after a moment's silence he said, "Now, Dwar Ev." Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel. Dwar Ev stepped back and drew a deep breath.
"The honor of asking the first question is yours, Dwar Reyn." "Thank you," said Dwar Reyn. "It shall be a question which no single cybernetics machine has been able to answer." He turned to face the machine. "Is there a God?" The mighty voice answered without hesitation, without the clicking of a single relay. "Yes, now there is a God." Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch. A bolt of lightning from the cloudless sky struck him down and fused the switch shut.