...but in a technical context, not in a scientific one.

One thing is identifying (age of copper; age of bronze) the best ways of smelting ore to obtain the metal through trial and error, another is to try and understand the nature of materials.

Those steps are both necessary and usually take decades to build understanding.

But the activity of "attempting to find explanations" is specific.

The attempt to formulate laws (the "better faring hypoteses") can bring to "universal laws" or to more humble "patterns". It will contribute to the scientific effort - it does not exhaust it.

Next time we get the urge to complain that GPT-n is just applying patterns seen in the training corpus we should remember humans don't do much better. We are just language agents with rich feedback from outside. We can't even write software top-down in one go without running the code.

> patterns

Except we consciously (intentionally) work on those matchings.

> are

Opinions should be offered in proper places. Polling is not debating.

Incidentally, that opinion is also offensive. Not just some tossed "All X are Y, goodnight", but a loaded one.

If you happen to conjure thoughts and let them roam wildly, well don't. Make an effort. Critical thinking as a conscious intentional effort must have plenty of literature. The "feedforward" output, the hallucination, is not a dish for service.

Corporate wants you to find the difference...

That is to say, the two things you mention are the same process. "Identifying the best ways of smelting ore to obtain the metal through trial and error" is the easy part, when you get to pick low-hanging fruits in a field. But as the easy options get cleared out, continuing improvements requires increasingly complex, sophisticated methods - that's where the process transitions towards "trying and understanding the nature of materials". It's still the same process, but couple layers of abstraction up from the original "how to get better stick than neighbor and beat them instead of getting beaten".

The difference, while moving in "interdependent" directions, is in the purpose: obtaining some sufficient information on how things work versus an actual consideration of the nature of things.

It is not really (fully the same process), because you could (in theory) "early stop" when you have achieved technically sufficient competence - the description of the optimal process -, before the jump to the understanding. Something works - that is the technical side; why it works is the scientific side.

Note that the approach has controversial sides: if you took Newton's "hypoteses non fingo", He refused that jump explicitly:

> hypotheses, whether metaphysical or physical, or based on occult qualities, or mechanical, have no place in experimental philosophy

I.e.: Newton proposed an «experimental philosophy» which stopped short of a "true understanding of the reasons behind phenomena", in order to avoid contexts in which solidity (at his time) could not be expected. "Understanding" as "identifying universal laws" was the best that seemed affordable.

I wanted to address that in the second paragraph, which I ultimately deleted before submitting, because I couldn't phrase it right. But since you brought it up: I'd consider this an effect of specialization.

In the process of improving your object-level "how things work" goals, you end up generalizing and stacking increasingly complex theoretical models. Soon enough, you end up with people working high up the stack - not knowing or caring about the initial goals. Those people end up growing the "mound of knowledge" both upwards and sideways. The work is sort of self-justifying, but really, it's also self-similar. Where early materials science may have been driven by, say, desire for better/cheaper weapons, soon enough, you have people doing materials science because they desire to solve a puzzle. Whether the practitioners are smelting different combinations of ores to find one that will win them the war, or they're mixing up different kinds of equations to figure out a clean solution to a theoretical conundrum - it's the same process, same motivation. And it always involves play.

The kind of methodical, boring approach, with hypotheses and control groups and peer-reviewed papers? That's the boring part you have to do after play.

See also (with no implied judgement in this context): software developers that lose sight of (or care little about) business goals, and instead aim for theoretical markers of what "good code" is, and/or solve abstract puzzles of algorithms and architecture. Or the MBAs that view companies as abstract money-printing processes, running them by the rulebook that's entirely independent of whatever it is the business is actually doing or selling. Both are cases of growing complexity creating a new field of work that's independent of what brought it into existence.

> an effect of specialization

It is also a specific purpose, a deliberate endeavour. We are in front of a world, we have to digest its phenomena, we try to understand it - it is a primal process, i.e. mental digestion. In some ways it could be argued that the "scientific drive" is anterior to the technical one: first you assess, then you act. Consistently, science started as "natural philosophy".

It is relevant, in the framework of "scientific drive as (development from the natural process of) mental digestion", that terminologically "science" is not just "the realm where false statements start to exist (and try to be avoided)" ("discriminate" - ex scindere), but also, before that, an activity of sorting, of "separating" conceptual elements - mental digestion. The "scient" (as a participle of scire, used as "to know") is a "distinguishing" mind.

We not only «desire to solve a puzzle» - it is our nature (as per the above) as minds dealing with a phenomenical world we need to grasp, as part of the activity of "seeing" it, already before action.

«Play[ing]» is a process that both allows to refine knowledge of the world, physically (in the physical play), and to refine the grasping of the world, mentally (in the mental play).

> growing complexity

Or just abstraction, enabled by a refined framework (following which, other areas become primary, topical - in the synthetic effort which is part of the above said process).

That's a necessary but insufficient condition for something to be considered scientific.

That's a common interpretation of what science is, it largely ends up being driven by confirmation bias. Because how do you know your hypothesis and test are even really connected? Or what you're seeing is a cause and not a correlation? This is why you need arguably the two most important factors in "real" science: predictability and falsifiability.

Predictability means that if your hypothesis is correct then you'd be able to formulate other improbable (and ideally currently untestable) predictions from it. And falsifiability means that if these predictions fail to occur then your initial hypothesis was also almost certainly wrong. So for instance Newton's hypothesis was that gravity was driven by a mathematical relationship between the mass/distance of two bodies. It was good science because it lead to the shocking ability to be able to dramatically simplify orbital dynamics, and create a complete predictive system of these bodies. It was even used to mathematically discover a completely unknown planet - Neptune.

Incidentally, his theory would also be able to be shown to be false if any of these unexpected predictions ended up being false. And that's actually exactly what happened. Observation of Mercury's orbit about the Sun showed it was off by ~1/3600th of one degree per century, relative to what was expected. And it's from there that people knew there was a mistake, which would only be explained later by Einstein hundreds of years later, who hypothesized a system with far more absurd predictions... and so the story continues.

ML has both of those things. In fact the situation is better than in many other sciences because the evaluation is completely digital. When you run an ML experiment you get back an objective measure in minutes or hours telling you if the model performed better or worse. If it's worse, boom, falsified. Many current papers are making predictive testable claims -- that based on various things we know what ideal parameter values are, the types of problems current NN archs will perform well on or poorly on, etc.

We've got lots of datasets at this point in history - it ends up being pretty reasonable to formulate a hypothesis, propose a change to the ML system, then see how it does on a host of datasets. Most papers focus on one or two datasets, getting some initial improvement going from train-to-test, but the successful ideas end up transferring to a wide range of contexts. The extension to other dataseets + contexts constitutes the prediction of performance on other systems.

And sometimes (often?) techniques do fail to transfer! IME, it's only the simplest ideas which do transfer to new contexts well - there's a lot of dark heat in pushing Imagenet another hundredth of a point, which you realize when you try to take incremental 'new SOTA' papers and apply them to audio problems.

But the things which /do/ transfer constitute real advances. And that's (drumroll) science! Sometimes things look good initially and get falsified. Medicine is still science, even if drugs get disqualified in phase iii trials...

It is not the scientific method at all. You didn't even include an analyze or a publish step, two critical components to the scientific method. The point is that science is a thoughtful, methodical, recorded, repeatable process that is scrutinized by not just your friends but by the entire world, including those who compete against you. In science, it's normal for your competitor to recreate your experiment exactly to see if you did it as well as you claim. Or to push it further.

What OpenAI is doing doesn't even resemble science at all.

Umm.. no. I’m pretty sure when science was “invented” (whatever that means), there was no way to share your research with “the world”. You wrote it down in a book in you cared to reproduce it.

Stop gate-keeping science. Observe, Predict, Test, Repeat. That is science. Kids do science when they watch adults do things. Just because you aren’t sitting in a lab publishing things no one reads doesn’t mean you can’t “do science.”

Just simply learning something isn't "doing science." The scientific method is a natural learning process it's not the only thing that defines science. This is incredibly light gate keeping. It's just maintaining a basic professional definition.

If a child watching an adult is "science", then when I brush my teeth I'm a dentist.

Words have meaning and when you dilute them down this far the only result is the destruction of communication.

Words do have meaning.

den· tist ˈden-təst : one who is skilled in and licensed to practice the prevention, diagnosis, and treatment of diseases, injuries, and malformations of the teeth, jaws, and mouth and who makes and inserts false teeth

Brushing your teeth does not imply you are a dentist.

sci·ence noun 1. the systematic study of the structure and behavior of the physical and natural world through observation, experimentation, and the testing of theories against the evidence obtained. "the world of science and technology"

A child may not be a good scientist, but they observe, experiment and explore their ideas. That sounds like science to me. Observational learning is a very common way scientist begin to understand things.

Do not forget that with ANNs we are still dealing with technologies with obscure sides: function approximators where the original function, and what it means, is oftentimes not reconstructed.

Replication of the experiment, "verification" of the law, can still be within a largely preliminary side - of the steps that describe behavioural details, patterns over "protocols", without the explanatory side (that which cause understanding, e.g. enabling deduction).

The publish step of the scientific method always seemed out of place to me - it means you can't discover anything on your own, which doesn't seem correct to me.

However, to discover something new (not just new to you) does require contact with established knowledge. And if you don't publish, you can't be very sure that it really is "new"; and it does not become part of established knowledge - it has not been discovered.

i.e. the concept of discovering new knowledge means it cannot be done on a desert island.

I think the colloquial understanding of "science" means studying how the world works. This is more like engineering: exploiting our understanding of how the world works to build new capabilities that we don't fully understand.

I don't think you would call it "science" for a bunch of single cellular organisms to cooperatively evolve a multicellular one. Similarly you wouldn't call it science when humans create digital lifeforms that require actual science to be done to understand how they work.

The reason that this is Engineering as opposed to Science, is that the hypothesis is just, "hey, maybe this will work".

Nobody has really explained why it works.

Which is a fundamental flaw in science.

Most of AI academics have spent their career theorizing complex algorithms or complex explanations of intelligence.

But the engineers have built large enough Neural Networks to give us data points that show intelligence is emergent out of relatively simple components.

Unsurprisingly, the people who believed they were the smartest were the least likely to explore the possibility that human intelligence isn't general, but specialized.

Echos of the academics building heliocentric models of the universe centuries ago.

Ask Feynman why magnets works... And he'll say no one knows. https://www.youtube.com/watch?v=36GT2zI8lVA

It's the same in ML. Good predictions do come from real ideas about how the systems work - ideas in information theory, entropy, cognitive science, statistical mechanics and so on - formulated in a context of our existing understanding and prior results. That's science.

Feynman was a genius. It takes a genius to say something like that, and it's surprising how many "scientists" are upset by statements like that, because they are so proud of their hard-won capabilities in modelling. Their curl and divergence operators, their Maxwell's equations... they forget that the map is not the terrain, and the universe is, at it's root, irreducibly mysterious, and the fact that we understand anything about it at all is gobsmacking.

>irreducibly

What are you basing that on?

This is my opinion, so take with a grain of salt:

I believe it's entirely possible that the scientific method breaks down past a certain level of system complexity defined somehow by thermodynamics.

This would in part be due to the infeasibility of running the proper experiments to understand the effects of single variables when tens of thousands of variables might be changing at the micro level.