In the industry, (preventive) maintenance takes up a pretty huge chunk of resources. It's something techs need to do often, and it's often a laborious task, but it's obviously done to reduce downtime.

So the business insight, as they like to call it, is to reduce costs tied up to repairs and maintenance.

All critical applications have multiple levels of redundancy, so that a complete breakdown is very unlikely, but it's still a very expensive process if you're dealing with contractors. If you can get techs to swap out parts before the whole unit goes to sh!t, then that's often going to be a much cheaper alternative.

But, in the end, it comes down to the quality of data, and the models being built. A lot of industrial businesses hire ML / AI engineers for this task alone, but expect some magic black-box that will warn x days / hours / minutes ahead that a machine/part is about to break down, and it's time to get it fixed. And they unfortunately expect a near-perfect accuracy, because someone in sales assured them that this is the future, and the future is now.

Let's define an "insight" as "new meaningful knowledge", just for fun. We could talk about what comprises "new" and "meaningful" but it would be beside the point I'm making.

In a supervised learning problem, the range of possible outputs is already known, meaning the model output will never be categorically different from what was in the training data. The knowledge obtained is meaningful as long as the training labels are meaningful, but it can never be new.

Unsupervised learning doesn't have a notion of "training data" but that means an unsupervised model's output requires additional interpretation in order to be meaningful. It is possible to uncover new structures and identify anomalies in new ways, but this knowledge isn't meaningful until someone comes in and interprets it.

Applied to the specific example where sensor data is used to try to generate insights about machine functionality: Either you can only predict the types of failures you've already seen, or you can identify states you've never seen but you wouldn't know whether they mean the system is likely to fail soon or not.

It's the Roth/401(k) tradeoff. For model output to be useful, someone must pay an interpretation tax. The only choice is whether it is paid upon insight deposit or withdrawal.

Yup, this is something I've seen from both sides. First you mention is basically the standard, while the last is part of the deep learning voodoo black magic that executives and sales love.

I've had people approach me with proposals like "What if we just churn [ALL OF] our data through this or that model, and let's see if it comes up with some patterns we've never seen or thought about"

And that's not just for industrial applications. It's everywhere.

What is concerning to me is that this mentality will surely induce more unrealistic expectations. Before you know it, business execs are starting to ask why we need business analysts at all, because surely those fancy deep neural networks can extract all kinds of features - "only need data scientists to figure out those things".

So yeah, that's my fear. That businesses will blindly start to discard domain knowledge, and just feed black-box models their data, and let the data scientists wrestle with the results.

Knowing how the outputs relate to the inputs is where most new insights could come from.

For example, what feature (input) is driving the “failure” of the machine (output/prediction)?

This is where ML explainability comes in.

AlphaGo is a supervised learner that outputs optimal Go moves given opposing play. It yields new discoveries in the same sense that a model designed to predict mechanical failures from labeled sensor data would: I didn't know what the model was going to predict until it predicted it, and now I know.

But what the factory owners want is a machine that can take raw, unlabeled sensor data and predict mechanical failures from that. They want insights. "Why not just feed all our data into the model and just see what comes out?" they ask. "I don't see why we need to hire at all if we have this neural net."

The reason you need a human somewhere in the system if you want insights is because someone needed to program AlphaGo specifically to try to win at Go. At the factory, someone needs to tell the machine what a mechanical failure is, in terms of the data, before it can successfully predict them.

Then, neither "winning at Go" nor "mechanical failure" are states that the system hasn't already been programmed to recognize. That's what I mean when I say a supervised learner cannot generate "new" output.

That's not what they wanted.

What people are being sold is AI/ML as a magic bullet that will do something useful regardless of the situation, and it lets business people avoid making decisions about what they actually want, because AI/ML can be anything, so they just signup for it and expect to get 20 things they didn't know they wanted handed to them on a plate.

Turn out, it's not enough to just collect a bunch of data and wave your magic wand at it. It wasn't with web analytics 10 years ago, it's still not.

What you actually need is someone who has a bunch of tricks up their sleeve, and has done this before, and can suggest a bunch of Business Insights the business might need before they start building anything, people that actually decide what to do, and actions taken to investigate, and solve those problems.

I mean, to some degree you're right; perhaps ML models could be useful for tracking hardware failures, but that's not what the parent post is talking about. The previous post was talking about just collecting the data and expecting the predictive failure models to just jump out magically.

That doesn't happen; it needs a person to have the insight that the data could be used for such a thing, and that needs to happen before you go and randomly collect all the wrong frigging metrics.

...but hiring experts is expensive, and making decisions is hard. So ML/AI is sold like snake-oil to managers who want to avoid both of those things. :)

It's all about how you package things. ML connected to an audio sensor could predict failure modes that are dificult to detect otherwise. Now that might not be was was asked for, but a win is a win.

I’ve experienced what the OP was alluding to...namely, it helps tremendously to start out with an understanding of the problem you’re trying to solve, more so in supervised learning. It’s incredibly frustrating to ask managers what business problem they are trying to solve only to be met with “We don’t know, that’s what we want the software to tell us.”

On the other hand, if they say “we want to predict machine failures” or “reduce maintenance downtime” now we have a lens in which to view the data.

If AI could do the magic as those managers hoped, they would be out of a job

Architectures are also tweaked slightly although they continue to be largely based on the two architectures that started the current deep neural net boom, Long-Short Term Memory Networks (LSTMs) and Convolutional Neural Networks (CNNs), which are largely used for natural language processing and machine vision respectively.

Every once in a while there is a bit of excitement about results that come out of adding a new parameter which is then given a fancy name like "attention" and generates much breathless copy about how neural nets will soon become as intelligent as a dog/ your three year old/ your personal assitant etc. Usually though it's not clear whether these new parameters or architectural tweaks are really responsible for the success of the proposed techniques and when one looks at the field as a whole it becomes more and more apparent that the most successful work is backed by the most data and the most computing power.

To slightly er tweak your prediction, I think what'll happen is that, indeed, like you say, there will be libraries and APIs and so on (well, there already are) but all these will be controlled by large companies that can afford the ground work. So, I don't expect small outfits, research teams at universities or tiny startups, to make a big difference. Academia is fast losing the ability to produce work that beats the state-of-the-art anyway (that's from personal comms with other researches). So in terms of academia perhaps we should expect a shift away from neural nets, to something that's easier to do research on.

That makes me particularly sad. In a utilitarian sense, the ideal situation would involve new approaches being democratised. Instead, we're ending up with walled gardens.

The current situation isn't dissimilar to what has happened to the Web Platform during past 5-10 years. And, similar predictions are made wrt VR and AR (both seem to attract different types of investors).

You train it with a lots of examples of "given this input, this is the output i want", and hopefully it learns to get the "correct" (similar input => similar output) output for new inputs that you feed it. i.e. you've now automated the process of figuring the correct output for a given input.

There is also the "reinforcement learning" AI paradigm where the trained AI is choosing actions (from a given repertoire) in order to maximize an action outcome score based on some scoring function you provided. This is appropriate in a situation where you want the "AI" to do something more than just select the correct output, but again no magic - you're having to anticipate and score the potential outcomes.

The typical AI methods train a model that utilises features that don't make much sense to humans (pixels/texture/word tokens etc). In contrast with traditional statistical modelling where each feature were given 'meaning' and their importance explored via investigating the observations (data) using properties of well studied mathematical models, the AI methods that often made the press don't put a lot of emphasis on these properties. The advancement is a push to utilise all available information/data to outstanding 'performance'. The explanation of the inner features within that math space is usually secondary (though I don't mean that authors of novel methods don't care about mathematical modelling).

I might be too naive here but that's how I feel after trying out many methodologies in my field.

Neural nets are mostly dominant in perceptual domains such as image or speech recognition, where the raw inputs represent a uniform sampling of data values (pixels, audio samples) over space and/or time. For classical business problems where the inputs are much richer and more varied, and already have individual meaning, then decision tree techniques such as random forests may be more appropriate and do provide explainability.

I mostly agree with that, however there have been advancement in scientific areas such as chemistry and biology (DNA/RNA) where the data are definitely meaningful and a lot times categorical. So the methodologies can be applied in wider areas, they just need a lot of domain knowledge and experience.

Except - and this rather ties into your point - those are not the only possible digits; your network also has to deal with (ie reject) other possible digits such as "P", "E", "3̸̶", or "[Forlorn Sigil of Amon-Gül redacted]"[0], which look like, but are not, decimal digits.

0: https://www.youtube.com/watch?v=ajGX7odA87k

If you only train it on examples of 0-9 then those are the only outputs it's going to give. If you fed a "P" into such a net then the outputs would be the degree of similarity of that "P" to each of the (0-9) digits it was trained on. You could of course threshold the output and ignore any prediction with confidence less than, e.g., 90%.

If you wanted the net to do a better job of rejecting non-digits, or at least some specific ones, then you could include a bunch of non-digit examples in your training data (so now your net has 11 outputs: 0-9 and "non-digit"), then hopefully - but not necessarily - it's highest confidence prediction will be "non-digit" when presented with a non-digit input.

Exactly.

> 11 outputs: 0-9 and "non-digit"

IIRC, this doesn't work (or not well) because the net tries to find similarities between the various members of the set "everything except these ten specific things", but you could just require low confidence for all digits on non-digit inputs as part of the gradient descent function.

The problem is more that - if people with decision-making authority trust the AI to not be insane and evil by default - failure modes like this have to occur to you before the AI starts misbehaving in production.

Isn't that what unsupervised learning is for?

For example, you might think of data clustering as an unsupervised problem, but in reality you're still providing supervision in terms of what similarity measure to use and some control over how many clusters it should generate; and, at the end of the day, the output is always going to be a set of clusters, not, say, a stock tip or wry comment on the nature of your data!

Asking for a friend.

So if they can solve the problem the ML probably can too ...

Things may have changed over the past 5 weeks or so with GPT-3.

Sure it's cool, but what problem is it solving? As far as I can tell the only useful function it has is polluting the internet with pseudo-intellectual comments to promote some agenda (likely political). So now that I think about it, it actually would be incredibly valuable for things like subverting democracy.

I'm still waiting for AI to figure out self-driving cars, natural language processing, and other "holy grail" problems that researchers have been chipping away at for decades but we are still a long way from having "solved" them.

Also in the case of self-driving cars, there is actually a very serious problem of it being difficult to guarantee safety. You can't tie every input to every output to demonstrate how it will behave (it is way, way, wayyyyyyyy too big for that), so you have to show statistically by driving so many miles that it's better. Which is still sound from an engineering perspective! But it leads to things like Teslas driving into trucks parked sideways across the road because the engineers simply never predicted it would ever happen. And now it's happened multiple times!

0. http://cyberlaw.stanford.edu/blog/2013/12/human-error-cause-...

What is "meaningful"? Honest question. Isn't meaning assigned by a reader? If I'm reading poetry generated by GPT-3 and I like it just as much as poetry written by a human poet, does it make it meaningful? What if I finetune GPT-3 (or the bigger next gen version) on every scientific paper ever written, and as a result it generates a novel idea that turns out to be valid and useful, should I care if it happened by accident, or without "understanding"? Can the knowledge encoded in the model parameters be interpreted as some form of understanding? If no, why not? What's missing exactly? What is different from how human scientists operate?

There's also the kind of meaning that scientists and researchers talk about when they extract knowledge from data. That's pretty different from communication; it's more a process of internal generation of notions and explanations that could later be conveyed in communication.

And then there's the meaning that is even more internal. E.g., reading tarot cards or tea leaves, people generate meaning out of nothing. And then there's Pareidolia: https://en.wikipedia.org/wiki/Pareidolia

If you're saying machine-generated text is meaning in the sense of that last category then sure, it's something we loosely call meaning. But it's fundamentally the same as other sorts of cleromancy [1], just more elaborate.

[1] https://en.wikipedia.org/wiki/Cleromancy

How is this "knowledge extraction from data" process different?

It seems none of this is something new. It's just that you no longer need a good education to learn about old ideas; you can come up with them on your own.

You said above: "There's also the kind of meaning that scientists and researchers talk about when they extract knowledge from data."

So if we agree that in both cases some form of knowledge extraction is happening, I wonder how these two forms compare.

How is this knowledge encoded in the heads of people? Perhaps through the choice of synapse strength and connectivity between neurons?

What is being encoded in GPT-3 weights as it reads through millions of pages of text? How is it different from tuning biological synapses?

Still don't know what I'm talking about? It's OK, neither do I :)

The way that something like GPT-3 tends to fail basically is that you 2-3 paragraphs where paragraph 3 will containment statements that subtly contradict the semantics of paragraph 1. Things like seeing things inside closed drawers and other cues that the thing has no fixed world-model. That's what gives an impression of "meaningless" or "no understanding". (admittedly, I've only played with GPT-2 but this is a description of how texts written by these models at first seem plausible and then implausible as you read more).

It's not some philosophical objection akin to "nothing but a human brain can think".

What is "world-model"? What makes you think GPT-3 does not have some kind of a world model? It's clearly not a very good one, but at the same time it does not mean it can't get better. A 3 year old also does not have a very good world model, what's the difference between his world model and one of GPT-3? Again, clearly there's a big difference, I'm just not sure we know enough about what's going on inside 175B model to make any dismissive statements about it. I'm also not sure what will happen if you train a much bigger model on much bigger data. Things might start to emerge at some scale.

You could argue that general intelligence is also based on prediction, and that a human's world model therefore isn't so different in nature, but there are some very significant differences ...

1) GPT-3's model is based only on a corpus of text (facts, lies, errors, etc) it was fed... there is no grounding in reality.

2) GPT-3 is only a passive model - it's not an agent that can act or in any way attempt to validate or augment it's world model.

3) GPT-3 is architecturally a language model .. it can get better with better or more data, but it's never going to be more than a language model.

The difference between a 3-year old's brain and GPT-3 is that the 3-year old's brain is not a one-trick pony ... it's a complex cognitive architecture, honed by millions of years evolution, capable of performing a whole range of tasks, not just language modelling.

The 3-year old's brain also has the massive advantage of being embedded in a 3-year old autonomous agent able to explore and interact with the world it's world model is representing... It you tell GPT-3 pigs can fly then as far as it is concerned pigs can fly, whereas the 3-year old can go seek out pigs and see that, in fact, they can't.

> if you train a much bigger model on much bigger data...

Yep, no one is denying that the potential exists for something to be created that can do something useful.

However, the point being made is that GPT-3 doesn't do anything particularly useful.

The generated text from is not consistent, and apparently fine tuning can result in poor performance (ie. generates random crap). It scored better at a bunch of metrics, which is great, but of questionable practical value.

"world model" or not, it's currently interesting, amazing... probably not useful unless you're writing a spam bot.

What is unclear, is if bigger models will actually solve that or not.

It seems to me the simplest way to decide whether or not something is meaningful (in this context) is whether or not the author (which is the algorithm GPT-3) can respond to criticisms against its own argument in a coherent way. In which case it has to get lucky twice, so it's that much less likely to happen randomly. If the author cannot respond to comments in a comprehensible manner, it's hard to defend the author.

I'm not sure about GPT-3, but let's imagine GPT-4 next year will be able to do this. It just does not strike me as a particularly high bar to clear. Let's go further, and assume GPT-5 in 2022 will pass the Turing Test (you personally will not be able to tell). What would you say then?

p.s. I built my first language model (LSTM based) back in 2014. Then I built a VAE based one. Then a GAN based. None of them were especially good, so I switched to music generation (this actually works pretty well). My most recent project is using sparse transformers for raw audio generation. Building novel DL models is literally in my job description.

Please don't wildly speculate about strangers you meet on HN.

The Turing test hasn't been passed if I'm the judge. Supposing I were unable to tell the difference between any AI system and a human interlocutor defending the same argument, at any time before 2023, I'd admit my prediction was wholly incorrect.

I doubt very much I'll find a taker for this bet, however. The AI field has always been bigger on optimism than results, as we both know.

However, I believe the models will be good enough to fool you during, say, a 20 question/response dialog. They will definitely be able to fool vast majority of unsuspecting humans. And they will definitely be able to keep track of conversation (remember what you said previously, and use it to construct responses to follow up questions).

Between the vast majority of unsuspecting humans and me there is a considerable gap. Mind the gap!

The TT ready model I'm envisioning will be trained on many billions of chat sessions. It will contain dozens of preconstructed graphs and will dynamically construct dozens more (personality graph, common sense knowledge graph, domain specific knowledge graphs, causality graph, dialog state graph, emotional state graph, etc), it will have a bunch of emotion detectors, humor detectors, inconsistency detectors, lie detectors, praise detectors, etc. It will have the ability to query external sources (e.g. google search --> web page parsing --> updating relevant graph). All these modules will filter, cooperate, and vote, providing input to higher level decision making blocks. These blocks will use those inputs to condition and constrain response generation process. This is finally where a language model comes in, and this until recently has been the hardest part - generating a coherent, grammatically correct, interesting text, directly addressing a specific prompt. This part has been solved. Until GPT-2 last year we simply could not generate high quality text. Now we can, and GPT-3 is even better at that. Sure, there are plenty of non-trivial problems left to solve, but I don't view them on the same level of difficulty - some of them have already been solved in the process of IBM Watson development, so I'm optimistic. The hardest remaining challenge is probably constructing common sense graphs. [1] looks promising.

p.s. your questions are so naive I'm not sure if you're trolling me. A human might answer them like this (and a bot built 50 years ago could easily imitate that):

"when and where were you born, and how do you know?"

- [personality - redneck] I was born on a farm in Oklahoma. How do I know what?

"what color, besides red, best communicates the flavor of a strawberry, and why?"

- Red is the right color for strawberries.

"What historical figure does my communication style make you think of most, and why?"

- You talk like one of them big city hipsters.

"Which of your family members comes to your mind first?"

- My little bro Jimmy, we just went fishing together on Tuesday.

"What do you think the context was in which the following poem was written?"

- [depends on the poem] I don't get this poem. What is it about?

[1] https://arxiv.org/abs/1906.05317

To clarify, the common sense problem is a hard one. It is similar to level 5 autonomy driving. That will take a while to solve. But what we are talking about here is kinda like Waymo cars which can drive themselves in ideal weather at slow speeds in Arizona. So in 2.5 years I think the best chatbots will be as far from having common sense as the current Waymo self driving cars are from level 5 autonomy. Which is to say they will be pretty good.

Ideal weather at slow speeds, with a professional human driver throwing road-condition curveballs at you and challenging your responses? I like my chances.

Keep in mind that you would have to differentiate bot's responses from those of an old truck driver, a snobby philosophy student, a conspiracy theorist, a stoner, a bimbo following latest Kardashians news, etc. Lots of different personalities of real humans who can throw curveballs at you during a chat session. I hope you don't expect to only chat with bay area devs? :)

This is a completely wrong reading of the situation. Tesla engineers clearly understood it is impossible to determine whether a stationary object is actually on the street or just an overhead sign with the equipment that is available on current Tesla cars and therefore they simply ignore all stationary objects. Because Autopilot is not a self driving technology this is not considered a problem. If a Tesla with enabled Autopilot ever runs into a stationary object it is clearly the driver's fault. It's extremely predictable and since it's easy to blame the driver there is no need to fix the problem by adding the required sensors to Tesla cars.

Isn't it good enough, or very nearly, to generate fake news? And if you can generate something that fools a large percentage of humans, even for a second or two, you can sell ads. It would be solving a problem for anyone who can profit from it. I thought the people who developed it stated that it was too dangerous to release widely? Dangerous = useful to bad people, no?

Don't get me wrong. You can absolutely generate news articles with these text generation models. But if you look at examples of the "fake news" generated by the monster GPT-3 model, it's not really any different to junk churned out by supposedly respectable news organisations i.e. grammatically correct but filled with logical contradictions, false statements that can be checked in 30 seconds with a search engine and so on. Most people already learned which news sources are trustworthy and which are pushing an agenda. If those outlets replace their journalists with GPT-3 it won't make any difference. The readers who don't trust them will still not trust them, and the readers who just want partisan cheerleading will continue to be satisfied.

What I see is actually quite different. If the people who could detect fake news was practically infinitesimal then you'd see the vast majority of people having super high trust in the media. In fact most people don't trust the media, lots of polls showing that. Sure they may only fact check something occasionally (often by reading about a topic they happen to understand), but people aren't stupid. After they notice or hear about a few mistakes and observe they're always in the same ideological direction, they get the picture.