I'm a huge fan of Peter Watts' work, but to be fair that is at the very least a close-cousin to the Chinese Room thought experiment that's been around for a while.
The details might differ, but it's worth thinking about whether the implications make sense. Watt's certainly asks a lot of good questions which other orders have also asked - i.e. Accelerando[1] is very interested in the idea of "what happens to mankind when economic systems can no longer be participated in by recognizably human minds?" - taken out to the limits of known physics.
[1] https://www.antipope.org/charlie/blog-static/fiction/acceler...
^^^ You read the full-text here, I highly recommend it.
What’s interesting in the culture series in light of the AI alignment debates is that the culture aligns every mind to its task at “birth”. A picket ship, falling outside the normal moral constraints, is a very different set of personalities and morals than a GSV such as wisdom like silence
An interesting aspect of the Culture Minds is that they're described as hybrids of human-like / neural-net / organic intelligence and also direct numerical compute and storage capacity. They can both "think" and "compute", something few if any current AI models attempt.
A well-known limitation of current LLMs like ChatGPT is that they are atrocious at even basic arithmetic, whereas a $2 pocket calculator from the 1980s outperforms them in this respect!
I'd love to see an attempt at a hybrid architecture where the neurons are also connected to a numeric compute component of the architecture.
Think of parsing an expression such as:
let foo = 5 + 10
"let", " foo", " =", " 5", " +", " 10"
[1309, 22944, 796, 642, 1343, 838]
[LET, ID("foo"), EQ, NUM(5), PLUS, NUM(10)]
What if we lifted this restriction and made an AI embedding where every token had both an ID and a value? For literal text just set the ID to NaN or whatever.
E.g.: assume that "integer constant" is ID 1000, then an LLM might be fed this input instead:
[1309, 22944, 796, 1000, 1343, 1000]
[ NaN, NaN, NaN, 5, NaN, 10]
Could be interesting...
As a layman with respect to LLMs, I feel like Blindsight and its perspective of the chinese room puzzle is required reading. I wonder if people working in AI feel the same and I'd be really curious to know how they feel about the book.
I don't think it is described as non-sentient in the book, but only as lacking consciousness.
Consciousness not being required for sentience is one of the two major themes of the book (the other being that consciousness is an evolutionary dead end).
Still your example is very relevant. I very much believe that ChatGPT, for example, is not conscious. The question is if it has some amount of sentience.
Sentience is a power of consciousness which is a power of the body which is a power of the cells of the body.
There is no dualism other than in the imagination: one could imagine some fuzzy distant "forest" without thereby imagining trees -- so much for the imagination.
Similar to the book Solaris (there is a new English translation, and I recommend both the US and Russian movies).
Apologies, I have a bad habit of mixing up those words, despite supposedly knowing better... at a conscious level.
Huh.
I wonder if I write some things subconsciously?
Decoding animal communication using machine learning is a popular research topic. See the current issue of Science, for example: Using machine learning to decode animal communication (https://www.science.org/doi/10.1126/science.adg7314)
Speaking of… one scene in the book has humans communicating with an alien that appears to have “learned” human speech by training a non-sentient LLM on human ship-to-ship transmissions, and then “fine tuned” it to achieve the desired communication goal without ever actually understanding what the LLM is saying.
This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s! That’s proper science fiction, right there.
Back to the economic aspect: Several characters in the book had to “butcher themselves” with implants and enhancements to remain economically relevant in the age of AIs. It’s that… or you’re packed away in storage. Useless.
PS: Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”. You could use this to scare whales away from ships without understanding what specifically the LLM was singing to them!