What I think we're doing is, you know, forming concepts which are not constrained to spaces. I dont even really think our concepts, which are mostly just bundles-of-techniques, live within the space of the target object, nor its measurment space (ie., that of experience).
Eg., my "pen" includes being able to grasp a pen (indeed, the relevant motor skills are prior-to and necessary-for the abstract concept formation).
What you're talking about is a pixel-space projection of "pen" being able to function like the (animal, body-first) concept "pen". It doesnt.
Rather "pen" in pixel space is, sure, an "internal langauge" of a sampling of AllPixelPatterns concerned with pixel-space projections of "pen". This i'd call a "template", and I dont think it has almost anything to do with concepts/representations/etc.
What we are doing when we acquire motor techniques which produce "ways of sensing and moving" that eventually could become reified as the abstract "Pixel Pen" is really nothing at all like sampling PixelPenSpace and deriving a tempalte.
To see the difference, consider that our concept allows us to resolve ambiguities -- eg., if I think something might be a pen, I can go beyond one space of measurement (eg., sight) -- eg., move the pen, write with it etc. -- and thus return to the target space "Is Pen?" with sufficient confidence.
Once such "mere templates" exist, everything we do, isnt required. Indeed, a calculator can take over at that point; put it in a cupboard forever to repeat whatever thin inferences pixel templates admit.
This is incorrect:
You can swap encoders and final fully connected sections, then mildly retrain the network to substantially save on training effort — “transfer learning”.
Further, you can compose networks, eg image recognition on top of something extracting a structural map.
This implies that the structures they’re “learning” generalize to different contexts — with a bit of retraining. In much the way your knowledge of programming transfers between languages (swapping encoders) and tasks (swapping final connected network).
Your point seems deeply based on this incorrect understanding of DNNs.
All these properties follow from properties of the data which we have structured to produce these solutions.
We arrange the target domain of one dataset up, so that the target domain of another is aligned --- and in doing so, solve the only problem which requires intelligence.
Yes, the character-space structure of Y labelled stuff aligns with the pixel space structure of Y labelled stuff. And feature space templates can likewise be associated.
This is just more of the same. King +Woman isn't Queen --- it isn't anything. This isn't reasoning. And likewise PicOfKing+PicOfW isnt PicOfQueen.
This is schizophrenic pseudoscience --- it's actually the mind of a schizophrenic who obtains sequential thoughts just by non-semantic associations.
The latent structure being learnt in both cases is just the coincidences we rig in the Y domain --- Inhave no doubt they 'transfer', we wrote them in
You can show a DNN a large sample of photos of animals, retain the lower sections of the network, and then show it a large sample of landscapes — and it will be able to recognize those faster precisely because a well-trained network picked up on the latent structure of shapes, and can reason about shapes in its internal language for describing pictures. It’s acquiring genuine semantic information — and learning about new experiences relative to that previous acquired understanding.
We didn’t create that similarity of form in nature — and DNNs discover that in much the way we do: finding recurring, abstract patterns to utilize in our internal language.
Animals acquire robust techniques which are not brittle to these sorts of thin contexts.
We do so by not templating pregiven data -- but by coordinating our bodies in the world, and rearranging it, so as to determine how to regulate ourselves in response to it.
Only a teen reading cosmo would think King-Man=Woman. The reason we can even interpret this thin formalism is because of our robust understanding of the concepts that these character symbols name.
With these concepts we don't cross domains, we regulate our own imagination, and so on, so as to create representations which are thick across an infinity of domains.
Talking in terms of feature domains has it backwards -- it assumes they're available for measurement. We don't transfer thin templates across domains. Our representations aren't within a feature space --- they're techniques for self-regulation.
With these we can construct templates in an infinite number of different feature spaces -- ie., project by imagination from our techniques
Humans ability of conceptual modeling is entirely constrained by their perception. Humans can't conceptualize a tesseract. All of their conceptualizations are in some form related to their perception.
When you're thinking of a pen, are you thinking of the atoms, molecule compositions and all the quantum effects going on? No. Your mind is, just like ML, working from a template constrained by your perception (visual, haptic, auditory).
Humans are more advanced than ML, but they are nothing special.
The Symbolic is a radically simplification of all the complexities impossible to be fully sensible. Even though the simplification is always particular, contingent, and full of ambiguity (human languages) and often inaccuracy (Newton's laws vs relativity), without the simplification, without Reason, ML systems are probably like animals, eventually succumbing to the full force of the complexity of reality.
Perceiving the world isnt a passive activity in which facts strike your eyes. The world doesnt have these facts, there is no data, and nothing from which to simply "average to a template".
When light strikes your eye, there is no "keyboard" in it. Nothing in it from which derive even a template of a keyboard.
There are only templates in datasets we prepare -- and we dont prepare them by actually "encountering datasets in reality". Rather, we arrange reality and measure it so-as-to-be-templateable.
What animals have is the ability, with effort, to engage in this dynamic -- to arrange the world to make it knowable. It is this process of arrangement which requires intelligence; not "taking the average after it has happened".
There is nothing in the world to be "percieved" in the ML sense, as in: ready for analysis. That's an illusion we construct: we make "perceiable data".
It is our bodies, and our concerns, which make the world itself perceivable. The world itself is infinitely dense with infinities stacked on infinities. There isnt "data" to be templated.
And this isnt theorertical: no ML system will ever work. All that works is our data preparation.
There’s three approaches to seeing the connection:
1. If the manifold hypothesis is true, then the semantic information between the original manifold and reduced manifold will be in the topology of that map — and your TT is a relabeling of the synthetic topology description.
2. For the convolution comment, take a TT statement, view it as a CuTT structure, and draw the construction diagram of that “shape”. Project that into a matrix encoding and… convolutions are right-side type division! Eg, finding a submatrix that matches “Line(…)” in your overall matrix. But it’s right-sided division since you can only remove the “innermost” factor in a product that way. (There’s also a left-handed, but it’s less relevant here.)
3. The paper on ML via hyper plane splits to create data coverings also suggests that what we really want to know is the topological relationships between kinds of data. This is furthered by, eg, “animal” in word2vec ideally being the centroid of the open ball covering animals, as they’re embedded.
That isnt a bridge principle between spaces which measure different properties.
The low-D structure of any high-D dataset is only a "washing out" of spare measurements of the same properties. These arent different "property-spaces", ie., the low-D structure isnt a world one has bridged to via statistics (or grad desc, etc.).
The properties the formal structure of text captures aren't the properties of objects.. they're something like the social-vocab-coincidental usage of language users in how they "orient their words" around topics. If we change our usage, models which find this structure, become invalidated.
We language-users routinely change this structure, and robustly against misunderstanding, because we bridge to the relevant property spaces by being in the world.
Isn't there an implicit conclusion here that the extra dimensions of 'world being' would actually make the manifold hypothesis true, it's just missing data in the training model? Not to say it's trivial to fix, that missing data might be derived laws of physics, smell, shared world models of a society etc. but none the less I fail to see how conceptually any of that couldn't be reduced to data.
But that isnt intelligence. Intelligence is the solution to the problem of those "all possibles" being imposssible. Namely, intelligence is how animal's conduct themselves in the face of not knowing what the data is..
they do this by conducting themselves and their bodies in their environment -- they dont focus on the problem in the AI/ML/etc. sense... In this way they dont make decisions based on "data" -- they move according to self-regulatory goals.
The data which they encounter, in that movement, becomes useful for reasoning about the world. But its radically more accidental, partial, and speculative than ML requires.
DNNs end up encoding parts equivalent to type theories in their operations — extracting an “internal language” (in the TT/CAT) sense which models the data. In particular, convolutions are a form of extracting/deducing types from your input. A full DNN deduces a model of the data in which it can type the incoming data and then reason about relations on those types to decide the challenge.
What is it that you think you’re doing, besides forming a model from data in which you reason based on an internal language, which describes the data you’ve seen?
Your post comes across as ignorant of eg, the overlap between TT and DNNs, while simultaneously full of woo about how humans operate.