As a lay-person, I was persuaded by Roger Penrose's argument that intelligence presupposes understanding, and that understanding is not computational. I had also finished reading a bunch of phenomenology before I read Penrose, so I was probably primed for that sort of an argument to be persuasive.

The difficulty there is, how do you understand things? "You" are mostly a chemical process, and I don't think there's much non-computational about chemistry. Penrose, as I understand him, answers "quantum mechanics", which I think is (a) kicking the can down the road rather than answering the question, and (b) problematic in its own right---the experiments supporting Bell's theorem seem to imply that QM and thus "understanding" are inherently random, right?

I actually do agree with you while still finding Penrose persuasive. I don't know how earnestly he adheres to the "quantum perturbations in nanotubules" (if I've gotten the incantation right) for which he credits someone else as a good try at a solution, and I have the impression he holds it as an open question.

I think the appeal (or interest, to me) from the phenomenology angle, is that we understand things by being in the world, having needs and desires, and some agency to obtain them. We understand a hammer because we can imagine a need to drive a spike, we can see that it can be held in our hands and be swung to drive the spike, that we have hands and a body, and that we occupy space.

All of that made me wonder if simply executing a program is sufficient for emergence of intelligence. This raises the naive question of whether adding enough sensors and actuators to the program would be necessary, or have I merely added something unnecessary to the problem? Anyway, that's the kernel of doubt that I have that a computer program alone is sufficient for the emergence of an intelligence, that is, an agent with the capacity to understand.

But why think understanding is not computational? It certainly enables a lot of computational behaviors, its effects can be modeled by computation. What power does understanding endow a system that a priori is beyond computation to capture?

'Consciousness' is just a synonym for self-aware. Goldfish are not self-aware.

How do you know?

Another fish, the cleaner wrasse, can apparently recognise itself in the mirror, hinting at some for of self-awareness.

https://www.nationalgeographic.com/animals/article/fish-clea...

A being is conscious if and only if it feels like something to be that being. What you're referring to is self-consciousness. Goldfish are most surely conscious but not self-conscious.

Stating that I can imagine X to be Y is not a statement that X is Y, only that if X is Y then nothing changes about my model of either X or Y.

That said, I'm not able to verify from a brief search for consciousness and self-awareness that the two are synonyms. The two seem to be related but are treated differently from one another. Furthermore it's not even clear whether goldfish are conscious or self-aware. Seems like it's an open question.