> Or, to put that another way: if you knew what algorithm the AI would be using to discriminate the signal from the noise in your data, why would you need the AI? Just write that algorithm.
My point is that this isn't the same thing at all.
Say your problem is plant detection from mobile phone photos. I can understand everything about plants, and I can manually build a highly optimised data processing pipeline.
But I can't build a feature extractor that outperforms ResNet50. That's the key algorithm.
> If the data is regular—that is, if a model of its structure can be held fully in a human brain—then jumping immediately to Machine Learning, before trying to just solve the problem with an algorithm, is silly.
True, but no one has made that argument. This is specifically about using hyperparameter optimisation vs improving your data pipeline.
Er, yes, I did, in my original post. The form you quoted was me attempting to be more precise in rephrasing it.
My point—my original point, this whole time—was that applying an advanced “feature extraction” algorithm to a data source whose features are explicitly encoded in a lossless, linearly-recoverable way in the data—what we usually call structured data—is silly.
For example, there’s no point in using ResNet50 to extract the “features” of a formal grammar, like JSON. It’d just be badly simulating a JSON parser.
In fact, there’s pretty much no data structure software engineers use, where ResNet50 would give you more information out than you’d get from just using the ADT interface of the data structure. What features are in a queue? Items and an ordering. What’s in a tree? Items, an ordering, and parent-child relationships. Etc.
The only place where it might make sense to use ML when dealing with structured data, is with statistical data structures like Bloom filters. ResNet50 might be able to recover some of the original data out of a bloom filter, in est using it as a compressed-sensing tool, or (in the algorithmic CS domain) as a decompressor for a lossy, underdetermined compression codec.
——
My second point was that, often, it turns out that your data is structured data, even when you didn’t ask for structured data.
Some natural-world datasets are structured!
Example: the standard model of quantum chromodynamics describes a clean digraph of possible spin configurations. You don’t need feature detection when looking at LHC data. The dataset is pre-bucketed, the items pre-tagged, by nature itself.
But more often, what happens is that your data turns out to not be “raw” / primary-source data, but rather a secondary source that was already structured, enriched, and feature-extracted by someone else before you got there.
Scraping social network data? It’s already a graph, and it often already has annotation fields in the JSON graph endpoints describing the relationships between the members. If you don’t just stop and look at the dataset, you might think your feature-extractor is doing something very clever, when actually it’s just finding the explicit pre-chewed “relationship” field and spitting it back out at you.
——
You might not see the relation to a kD-sample-matrix feature-extractor like ResNet50, so here’s some more tightly-analogous examples:
• What if the images in your training dataset turn out to be in Fireworks PNG format, where the raster data contains an embedding of the original vector image it was rendered from? Specializing your feature-extractor to this data is just going to make it learn to find those vectors (and extract features from those), rather than depending on the features in the raster data; and then it’ll fail on images without embedded vector descriptions. And if that’s all you want, why not just use a PNG parser to pull out the vectors?
• What if your audio files turn out to all have been MIDI files rendered out from a certain synthesizer using its default set of instrument patches? Will feature-extraction on this rendered data beat just writing a program to exact-match and decode the instruments back to a MIDI description? Certainly there might be MIDI-level features you want to extract, but will ResNet50 be better at extracting those MIDI-level features for having seen the rendering, as opposed to having been fed the decoded MIDI-level data directly?
Ok. No one other than yourself has made that argument.
> Scraping social network data? It’s already a graph, and it often already has annotation fields in the JSON graph endpoints describing the relationships between the members. If you don’t just stop and look at the dataset, you might think your feature-extractor is doing something very clever, when actually it’s just finding the explicit pre-chewed “relationship” field and spitting it back out at you.
I've worked in this exact field, and I've never ever heard of anyone who doesn't do this. I guess someone might.. so if your point is that there are dumb people around then ok.
But that isn't what a feature extractor does! In the graph context a simple feature extractor is something hand coded like the degree of a node, or a more complex learned one is something that maybe does embedding.
If your point is that people should understand their data, then yes that is in data science 101 for good reasons.
Most of what you wrote seems fine, until I got to this. A query optimizer seems like something that tends to be very opaque, very complex, and in my experience blows up without a good explanation frequently in typical situations. It's also based on a lack of complete data about the problem domain, to the point where an optimal algorithm seems hopeless. I'm not saying an AI approach automatically can be better, but at least you (I) can envision it being better, perhaps less brittle and taking into account dimensions and possibilities a human doesn't. And the non-AI solutions aren't trustworthy in terms of bounded quality so you have not got a lot to lose.
I can imagine cases (distributed databases, different speed storage) where it would make sense to test the queries and learn which optimisations make sense. It'd be self tuning and able to adapt to changing hardware.
I spent way too many years writing ad hoc Oracle SQL that had to complete within a few hours and trying to guess if the optimizer had decided to finish in 15 minutes or a week.
And I would read Tom Kyte where he says obviously your database is set up wrong if the optimizer isn't working for you. And how you should do everything in one big beautiful query that uses all the latest features of Oracle.
Yes, you can. If, that is, you can actually understand what the produced model is doing. And, of course, no human can do that, because no human understands the algorithm being employed by the produced model, because it's a really freaking complex algorithm whose optimal formulation really is just a graph of matrix transformations, rather than an imperative procedure that can be described by words like "variable."
This is an important idea to absorb, for the specific case where the AI converges on an optimal algorithm that's actually very simple—because the data has a regular, simple shape—rather than on one that's too complicated for our mortal minds. If you already knew that simple algorithm, then the work you did training an AI just to end up back at that same simple algorithm is wasted effort. An AI can't do better at e.g. being an AND gate, than an actual AND gate can. An AI can't do what wc(1) does better than wc(1) can.
If the data is regular—that is, if a model of its structure can be held fully in a human brain—then jumping immediately to Machine Learning, before trying to just solve the problem with an algorithm, is silly. The only time you should start with ML, is when it's clear that your problem can't be cleanly mapped into the domain of human procedural thinking.
The AI programmers of the 1960s were not wrong to start with Expert Systems (i.e. attempting to write general algorithms) for deduction, and only begrudgingly turn to fuzzy logic later on. Many deduction tasks are algorithmic. If you don't require the context of "common sense", but only need operate on data types you understand, you can get very far indeed with purely-algorithmic deduction, as e.g. modern RDBMS query planners do. There would be no gain from using ML in RDBMS query planning. It's regular data; the AI's trained model would just be a recapitulation of the query-planning algorithm we already have.