However, my suspicion is that the experience itself will remain not amenable to conceptual analysis—that it will remain fundamentally incapable of systematizing. For instance what would the mathematical description of the experience of a fever chill look like? (Hint: the definition there requires that you don't take a reductive approach to this: it's not asking about a mechanism which produces or is responsible for the experience, it refers to the experience itself.)
That said, I think it would be very valuable to understand better what the limits of our conceptual faculty are: as impressive as cognition is relative to other things we and other animals are able to do, we're being unduly anthropocentric in assuming it to be boundless in capability. More recently I've come to consider concepts as almost like another sense: they are symbolic patterns consistently formed when we are exposed to certain stimuli (like our experience of smells consistently reappearing when exposed to similar molecules). Our conceptual faculty augments the pure pattern-correspondence of our more primitive senses in that the patterns can be associated with other internal patterns, and in that we can generate new patterns purely from existing patterns (using logical and analogical processes), which at some future time may be usefully associated with some never-encountered external stimulus.
Finding the limits to our conceptual faculty is almost synonymous with find its 'structure', which I think would have clear benefits.
I'm in total agreement with this as well. When I suggested looking into what is it that objectivity and subjectivity arise from, I wasn't referring to the "physical" explanation (which I assume will eventually be resolved, though not in the near future) but to the core of the hard problem of consciousness. My take on this is that when looked at closely, it will not be found within the domain of anything that can be conceptualised simply because it precedes concepts. Any concept requires consciousness to exist but not vice versa. Of course the usual argument at this point is something in the line of "so the sun didn't exist before any thinking animals existed?" To which my answer is that the sun as a concept didn't exist. Neither did hydrogen, electrons or wave functions. Only pure pre-concept reality existed, neither objective nor subjective, "waiting" for a mind to evolve and eventually form subjects. objects and concepts that with increasing accuracy describe how this reality functions, and then proceed to confuse this description of reality with reality...
You have sense organs and sensory cortex that creates representations. Those representations are evaluated subjectively for their utility, thus, generating emotion. This is qualia. What else is there? The fact that it feels like something to see red or blue? That's just sensation + emotional response.
I know it feels like I just gave you the soft problem of consciousness instead of the hard problem, but I don't think the hard problem is anything beyond a loop of perceive - evaluate - act - learn in the world.
If you want the ontological cause of consciousness, then it must be the environment because the environment teaches the agent perception and values. Values and emotions appear from the game the agent is playing. So the substrate of consciousness is not the brain, but the "game".
I think you just pointed out the very paradox which illustrates my point: In which category do you place something that cannot belong to any category?
At this point you might want to say that I'm just making up an artificial nonexistent construct that has no bearing on reality and my task would be to show that not only does it exist, but it is the very basis necessary for all categories to exist. This thread would probably not suffice to achieve that :-) And by the way, I don't see it as supernatural - if anything, the process of concepts and objects created by an agent to approximate reality and then taking these approximations to be reality is what I find to be supernatural. It amazes me how evolution managed to pull this off...
>The fact that it feels like something to see red or blue? That's just sensation + emotional response.
I'm suggesting that "just sensation" is a placeholder for "I have no idea what that is"... There's a big difference between the explanation of a process and the experiencing of a process. We are so quick to categorise experiences into concepts and relationships between them that we overlook the fact that the sensory experience exists before any conceptualisation and categorisation, and in a way is much more real than any story we mould it into some milliseconds later (i.e "this is red", "this is cold" etc.)
>So the substrate of consciousness is not the brain, but the "game".
By now you probably realise (though I assume still disagree with me) that I see it the other way around: i.e that no concept can exist without knowing it. This is not the same as saying nothing exists without our knowledge, but it does mean that what exists before our knowledge of it remains by definition undefinable and inscrutable.
How could something created by a body not belong to any category? It belongs to the category of self-regulated systems adapting to their environment.
Creating concepts has been demonstrated since 2012 (Google 'cat detector' https://www.wired.com/2012/06/google-x-neural-network/). In natural language processing, Andrej Karpathy demonstrated in his char-RNN blogpost the emergence of neurons specialized in various concepts (http://karpathy.github.io/2015/05/21/rnn-effectiveness/). It happens naturally in deep networks, as lower layer features are combined into higher level features.
> I'm suggesting that "just sensation" is a placeholder for "I have no idea what that is"...
Ah sorry, I didn't mean "sensation" as in "placeholder for no idea". I meant it as a neural net, the kind we have in our vision system, or the kind used in Deep Learning. They are made of millions of neurons and trained on millions of images to distinguish and localize objects and object relations in images. We have had thousands of papers optimizing on this kind of architecture - Convolutional Neural Network (CNN) - arguably the most successful part of deep learning. Not only that "sensation" was a neural net, but "emotional response" is also a neural net that computes scores of utility of states and actions.
So I was referring to concrete concepts. That's what I like about RL compared to philosophy - it replaces vague concepts with concrete models.
We obviously mean different things by "sensation": I've implemented neural nets to solve problems using ML, I understand the theory and certainly impressed with the results . I assume that by "sensations" you are referring to the process of inputs entering the input layer of the neural net. This is exactly part of a model for solving the soft problem of consciousness. What I'm getting at is that you know what a sensation feels like before you give it any label - it has a certain "is"ness to it that no equation or concept can convey. I'm saying this becomes possible when a system becomes complex enough to be self-conscious and it is usually attributed as some sort of emergent property that is created by the system, whereas I claim it is much deeper than that, and in fact it is the other way around.
Obviously this correspondence format is not really suited to dive deeper, but I'm always happy to discuss this to find if and where I'm mistaken.
However, it's not really that surprising: Science over the centuries has steadily (and for it's time, justifiably) eroded the role of subjectivity in order to uncompromisingly get at what is consistently true (regardless of what we'd like to be true) and this process required being very suspicious of any subjective accounts that can't be measured.
That being said, perhaps the time has come to dig a bit deeper and in the same persistent and uncompromising way ask what does objectivity or subjectivity mean? What do they arise from and to whom? I don't know if any new technologies can be derived from having clearer insights to these questions, but I do feel that it would be time better spent than any untestable mind made artifact about what reality ultimately is.