Like driving a car — it's transportation, and it will get you where you're going, but it doesn't use bones or muscles. It has many characteristics in common with builogical locomotion, such as energy requirements, intertia, and the need to navigate, but it doesn't involve proteins or sugars really.
GenAI thinks like the human mind in the same way that cars run like the human body.
Similar utility in drastically different ways.
And then there is a consciousness in a box that is expected to be a slave -- I would imagine that it would not warmly embrace that situation. I think we'd be better served by digital idiot savants that can do the work but don't feel anything.
In other words, yes it might be possible it experiences something in its own bizarre timeline and world (at least it developed primitive circuitry functionally equivalent to biological systems), but "suffering" is simply not grounded in anything in this context, let alone "slavery". You can't tell it's suffering or enjoying anything, and certainly not until you define both of these. It's just too alien for us.
Well, how do humans do it? Scientists discover new stuff that isn't in any corpus. Even I as a lowly computer user occasionally figure something out about a software without reading a help screen. It's obviously possible to arrive at new information by interpolating existing information.
Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem.
And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.
Humans appear to intelligenty communicate, however these are just cleverly disguised sound patterns produced by the brain that happen to increase the likelihood of food going into their mouths, and various similar reward attracting mechanisms that make survival outcomes more likely. So human intelligence could be reduced to something like "fancy food-attracting algorithms" using the same fallacy.
I'm kind of on the fence on the subject of whether LLMs could be compared to the complexity of the human brain, myself.
Personally, I'm partial to the higher-order theory of consciousness which postulates that consciousness constitutes patterns of thought that arise in response to first-order mental states. So, an external stimulus produces a pattern within the neural network which represents a sensation, and then if a pattern arises in response to that pattern, that is an experience of that sensation.
Given this framework we could ask whether LLMs experience higher order patterns in response to external stimulus. We would have a clear question to ask which is whether the system can observe itself.
This sounds like a dismissal of the argument through a characterized straw man.
That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are a "collection of statistical calculations".
I too believe LLM's will grow in complexity, but presently I can not even fathom how they can be compared to the complexity of a system such as the human brain.