We've drawn a couple of circles and we want to draw the rest of the owl, and you're telling us there's only one step.

Yes. Because we don't know it is an owl we should draw. It's equally likely AI is just three circles combined.

exactly... we don't even have a definition of consciousness that is better than Jacobellis v. Ohio - For that matter, I'm not sure we even have it that good. I'm not sure I'd recognize a non-human consciousness when I saw it. I'm not sure anyone would.

I mean, sure, I can totally see the idea that consciousness is maybe some emergent property of complex systems... in which case, sure, we could stumble upon it by accident. That is totally possible; we're creating complex self-replicating systems all over the place.

But as far as intentionally constructing the thing that will come after humanity? we can't do that until we know what, exactly, it is we are constructing, and as far as I can tell, we're pretty far away from any idea of exactly what that is.

Is consciousness a necessary component for AGI, though?

I don't think it's necessary for us to get a machine to reach "consciousness", whatever that is, before we realise it's overtaken our mental ability to combine two existing concepts in order to create a new one.

I imagine this is what's going to happen: we humans have drawn two circles (thank you dear fella for that analogy) and we are about to draw a third, combing two existing concepts.

At some point we will have drawn so many circles that the next one will not be drawn by us. It will be drawn by AI.

Who knows when we'll reach that level of machine intelligence and if it even requires them, the machines, to be conscious? But we will, without any doubt, reach a point where there have been so many circles drawn that they are now smarter than us at drawing them. Absolutely without any sort of doubt in my mind this will happen.

Tens of thousands of people comprehend Einstein's work. It used to be hundreds. We're smart as shit these days. Fuck me if it's not before I die.

>Tens of thousands of people comprehend Einstein's work. It used to be hundreds. We're smart as shit these days.

In some ways, I feel like this is the real singularity. This trend towards feeding and offering education to everyone. Like, it used to be that to be an intellectual, you'd need to be born rich to get the free time and education. I mean, being born rich still helps, sure, but a lot more people are getting a shot.

On the other hand, it's a process that levels off at some point, like everything else. The population is leveling off; as we bring education and leisure time to more and more of the world, eventually most of the people who have the capability to do this sort of thing will have done it.

I mean, I see that with computers, too... like with the leveling out of Moore's law; exponential processes, in nature, tend to not remain exponential.

If you subscribe to the theory of great and continuing, accelerating change due to technological improvements, advanced technological education for everyone is an expected ovservation during the decades or centuries it’s occurring :)

>Is consciousness a necessary component for AGI, though?

I always thought so. Like, without volition, it's just a really good natural language tool, and not a general intelligence.

What would an artificial general intelligence without consciousness look like to you?

Like the intellectual equivalent of a beast of burden - good at solving open problems by (apparently) intuiting connections and making inferences, probably with the appearance of creative operation and experimental learning, but lacking any genuine desire or personal agency.

IMO this is much more plausible as AGI than some kind of mythical general meta-intelligence that has a virtual soul.

I don't see why consciousness or agency are at all necessary for AGI - except as a science fiction trope.

Besides which, consciousness and agency are two different things, damnit! I've got a github repo in which I train agents by policy optimization. I have no illusions that my Pytorch models are conscious.

>good at solving open problems by (apparently) intuiting connections and making inferences, probably with the appearance of creative operation and experimental learning, but lacking any genuine desire or personal agency.

so, like a philosophical zombie? like something that appears to be conscious but isn't?

I mean, we could call it an "artificial general intelligence" or we could call it a "generalized optimizer & problem solver"

So basically if we can have a system that can solve any problem thrown at it, that's essentially the AGI.

A really efficient slave. Essentially it does not get creative and is ultimately predictable, even in failure modes. It would also have no subconscious drives to do unexpected things.

Capitalistic dream.

Asimov's Three Laws of Perfect Slavery (Forgive me Isaac!)

If such beings were conscious they would be saints.

One of my favorite AI jokes:

Q: What's AI?

A: When the machine wakes up and says, "What's in it for me?"

It would be fetish to actually make humanoid servitor robots. IMO.

without creativity... I don't think it would make for a very effective slave in a world that is constantly changing.

Is creativity possible without consciousness? I don't pretend to be an expert.

Probably possible, yes. Subconscious drives are a thing. You could have sentience without consciousness. (The difference is that consciousness can change or modify drives and recognize its own actions on the world - its own agency. It is related to sentience and the line between the two is thin and blurry.) I'm not sure it is possible to be conscious without having been at some point sentient.

The idea was that such an AI would never question its priorities or change them, not that it wouldn't be able to solve novel problems.

(Which is the definition of intelligence, duh. And shows how far we're from that. What we have is glorified pattern matching algorithms and some basic symbolic logic and clustering.)

That sounds more like a Communist dream. Capitalism doesn’t reward really efficient slaves.

I may not know an owl until I see it, but I do know that's not it.

Applying this method of thinking, is there anything that we aren't one invention away from?

Large scale colonization of Mars would require multiple inventions. We know this because we have a good understanding of what a successful colonization of Mars entails, and have a pretty good idea of the main technical obstacles. AGI, on the other hand, might be one breakthrough away or two hundred breakthroughs away. We just have no idea what makes intelligence tick.

I'm just trying to stay positive.

If you talk optimistically about AI (or big data for that matter) you quickly get downvoted these days. I'm curious as to why that is. I thought this was 'marica. Isn't this 'marica?

Edit: my apologies, that was condescending. I should have finished of with a wink of an eye.

There's optimism and there's being too vague to disprove. You might be able to dodge downvotes by adding more objective content to your comments.

History has played out a particular way quite a few times (https://en.m.wikipedia.org/wiki/AI_winter). If you can tell us why today isn't like all the other days using facts, I'll bet the downvotes will stop.

General intelligence? Does it even exist? I find the evidence that humans have it unconvincing, and I've seen nothing that makes me think we're "one invention" away. The direction of machine learning seems to be specialised tasks on specialised data sets; surely the very opposite of any kind of general intelligence.

So get whatever intelligence human have in computers. That was the goal. Call it general intelligence or call it x.

Human intelligence works well [0] from inside human shaped objects with human shaped biology, in human shaped societies, and to no small degree for doing human-style tasks.

I am not convinced that putting that human intelligence, whatever it is, into a silicon box will give us the results we hope for (or is even a sensible ambition, given the degree to which our intelligence is part of our biology).

[0] Actually, maybe it doesn't work really well - maybe it's a slow-motion car crash over millions of years. But it works well enough that we clearly want more of it.

Whatever human intelligence is, there's clearly some part of it that can do things like design computers and build houses. That part, at least, people would like to be able to match in performance using machines. It's 'intelligence' in the sense of being able to do things like think up clever strategies in novel environments that people want to put in a silicon box.

If that is the goal, it's certainly not what the current batch of machine-learning and hype-train riders are chasing. They (or at least, the ones doing it via huge training sets) seem to be working on the very opposite of novel environment situations. I wonder if anyone is seriously pursuing that.

That's exactly what they thought in the 1950s and 1960s 1970s and 1980s. And oh by the way, our brightest minds were working on it then too.