We can't produce arbitrary media streams with many "stack layers" of meaning and detail yet, but we can do a lot of specific instrumental transformations...
Vaguely relevant: https://koe.ai/recast/
Unfortunately this is not true. It takes a huge amount of human effort to make MIDI encoded music sound good. The difference between MIDI and raw audio music generation is the same as the difference between drawing a cartoon and producing a photograph.
To clarify, yes MIDI can be expressive, but what's being generated when people say "AI generates MIDI music" is basically a piano roll.
As a clasically-trained pianist who then got into electronica and synthesis, it was mind blowing to me that people could wrangle expression and phrasing from a MIDI sequencer.
The search space is absolutely enormous, though, so I don't dispute that it's very difficult, but I wouldn't go so far as to say that it can't be done. In such a space there are "no wrong answers" so to speak. I have a python script which creates randomized sequences of notes/rhythm and gives each one a different combination of LP/HP filters and random envelopes - it's not music but it takes on a much less mechanical quality by emulating different attacks and timbres over time, even though it's completely random.
I would go so far as to say I'd be genuinely surprised if algorithmic composition and production hasn't been used to some extent significantly greater than "basically a piano roll" in at least some of the past decade's top 40 music on the radio.
There is such a reason - lack of training data. Very few high quality detailed MIDI samples exist to train machine learning models like AudioLM.
For state of the art in MIDI generation, take a look at what https://aiva.ai/ produces (it's free for personal use). There you can compare raw MIDI output to an automatically generated mp3 output (using "VST's and samplers with routing and effects in place, then using some combination of genetic algorithms and other methods to "tweak the knobs" in the search for something pleasing.")
mp3 version will sound much better than raw MIDI, but (usually) significantly worse than music recorded in a studio and arranged/processed by a human.
Would we expect a discrete melodic structure to be expressible as averages of prior music? No.
This reads like someone who knows sheet music and theory but does not listen to music. It’s repetition of short phrases over and over.
I’m not really sure what people expect of general AI trained on human generated outputs. It can’t make up anything anything “net new” only compose based upon what we feed it.
I like to think AI is just showing us how simple minded we really are and how our habit of sharing vain fairy tales about history makes us believe we’re masters of the universe.
AIs state will forever be constrained to the limits of human cognition and behavior as that’s what it’s trained on.
I read published research all year. Circular reasoning. Tautology. It’s all over PhD thesis.
There’s no “global structure” to humanity. Relativity is a bitch.
Seeing the world through the vacuum of embedded inner monologue ignores the constraints of the physical one. It’s exhausting dealing with the mentality some clean room idea we imagine in a hammock can actually exist in a universe being ripped asunder by entropy.
It’s living in memory of what we were sold; some ideal state. Very akin to religious and nation state idealism.
"AI" is just taking `mean()` over our choice of encodings of our choice of measurements of our selection of things we've created.
There is as much "alike humans" in patterns in tree bark.
AI is an embarrassingly dumb procedure, incapable of the most basic homology with anything any animal has ever done; us especially.
We are embedded in our environments, on which we act, and which act on us. In doing so we physically grow, mould our structure and that of our environment, and develop sensory-motor conceptualisations of the world. Everything we do, every act of the imagination or of movement of our limbs, is preconditioned-on and symptomatic-of our profound understanding of the world and how we are in it.
The idea that `mean(424,34324,223123,3424,....)` even has any revelance to us at all is quite absurd. The idea that such a thing might sound pleasant thru' a speaker, irrelevant.
This is a product of i dont know what. On the optimist side, a cultish desire to see Science produce a new utopia. On the pessimisst side, a likewise delusional desire to see Humans as dumb machines.
What a sad state!
> The idea that `mean(424,34324,223123,3424,....)` even has any revelance to us at all is quite absurd.
Most of what I say to anyone is exactly this.
When I'm about to give anyone any information, I look back at all of the relevant past information that I can recall (through word and sensory association, not by logic, unless I have a recollection of an associated internal or external dialog that also used logical rules.) I multiply those by strength of recollection and similarity of situation (e.g. can I create a metaphor for the current situation from the recalled one?). I take the mean, then I share it, along with caveats about the aforementioned strength of recollection and similarity of situation.
This is what it feels like I actually do. Any of these steps can be either taken consciously or by reflex. It's not hidden.
> I think it's deeply depressing that AI has been sold as something even capable of modelling anything humans do
This is a bizarre position. All computers ever do is model things that humans do. All a computer consists of is a receptacle for placing human will that will continue to apply that will after the human is removed. They are a way of crystallizing will in a way that you can sustain it with things (like electricity) other than the particular combination of air, water, food, space, pressure, temperature, etc. that is a person. An overflow drain is a computer that models the human will. An automatic switch/regulator is the basic electrical model of human will, and a computer is just a bunch of those stitched together in a complementary way.
You're no more made of clay & god's breath, as you are sand and electricy.
You're an oozing, growing, malluable organic organism being physiologically dynamically shaped by your sensory-motor oozing. You're a mystery to yourself, and these self-reports, heavily coloured by the in-vogue tech are not science, they're pseudoscience.
If you want to study how animals work, you'd need to study that. Not these impoverished metaphors that mystify both machines and men. No machine has ever acquired a concept through sensory-motor action, nor used one to imagine, nor thereby planned its actions. No machine is ever at play, nor has grown its muscles to be better at-play. No machine has, therefore, learned to play the piano. No machine has thought about food, because no machine has been hungry; no machine has cared, nor been motivated to care by a harsh environment.
An inorganic mechanism is nothing at all like an animal, and an algorithm over a discrete sequence of numbers with electronic semantics, is nothing like tissue development.
What you are doing is not something you can introspect. And you arent really doing that. Rather, you've learned a "way of speaking" about machine action and are back-projecting that onto yourself. In this way, you're obliterating 95% of the things you are.
> these self-reports, heavily coloured by the in-vogue tech are not science, they're pseudoscience.
I simply don't know what you're referring to. If you're referring to retrieving memories through associations, there's mountains of empirical evidence for that. If you're referring to wondering if I remember things, and being unsure of the information I'm recalling when I have less recall of that, or wondering if past situations compare well to current situations, well you got me. It's my personal belief that conscious thought is an epiphenomenon that is a rationalization of decisions already made.
But the rest of this is nonsense. Vivid imagery is not an argument for exceptionalism, no matter how much I say things drip or ooze. This is just association in action. You're trying to create a distinction for life (or rather what you recognize as life) life oozes and has viscera, so using a bunch of words that feel wet and organy can substitute for reason contra the robots.
We're trying to train a full composer AI without allowing to learn about different instrument sections independently at first. The human composer will have a good idea of the different parts and know how to merge them in harmony.
I think we might get better results training separate AI systems on percussions, strings, vocals etc. then somehow create connections between them so they learn together. A band AI if you will.
We could try a BERT for each, with the generator learning to output logical sequences of sounds instead of words.
What's more interesting and concerning - listen carefully to the first piano continuation example from AudioLM, notice the similarity of the last 7 seconds to Moonlight sonata: https://youtu.be/4Tr0otuiQuU?t=516
I'm afraid we will see a lot of this with music generation models in the near future.
Having said that, I think the idea of predicting popularity is good - we can use it for filtering already generated chunks during post-training evaluation phase.
I don't think the other two methods you suggest would help here, we want to generate while conditioning on famous pieces, and we don't want to increase temperature if we want to generate conservative, but still high quality pieces.
It's true that we (humans) are less sensitive to plagiarism in the text output, but even for LLMs it is a problem when it tries to generate something highly creative, such as poetry. I personally noticed multiple times a particular beautiful poetry phrases generated by GPT-2 only to google it and find out they were copied verbatim from a human poem.
Another similar possibility might be to do more RL with this data, e.g. using upside-down RL. One can possibly steer this with user feedback as well.
It didn’t get to train on the test set, if that’s what you’re implying, and I find it hard to believe the assertion that continuations are copies of the train set (if that’s your claim).
I guess in retrospect we asked it to continue the music in a likely way, not be novel. And it definitely convinced me enough to be impressive. An NN that composes completely fresh music, whatever that means (I’m sure most modern human music has a hefty dose of cross song sampling), would certainly be a good next goal post.
but yes there is not yet at on-demand button rendering from a text prompt of bitstreams encoding composed performed and mastered music.
Reason derives its strength in having a few primitives and creating new assertions through the transformation of symbols by following precise rules (which is how algorithms work).
In ML-based AIs, everything is imprecise and probabilistic, and this kind of generation gets its strength from building recognizable from utterly imprecise inputs and training - quite the opposite of how logic and reason evolve. Now, "classic" AI was a powerful way to derive new knowledge, and automatic theorem proving is a strong discipline; but the recent breakthroughs in AI are not directly applicable to classic techniques.
Do you know what machine-learning AIs could be good for? Generating "insight" in problem solvers for guiding the theorem demonstrations through the proof search space, trying to find the best sub-spaces to explore. If there's a way to create human-like general AI, it will likely combine both kinds of generation - the rational methods of symbolic logic and the "irrational" statistical methods of ML.
Work in progress for sure, though.
How so?
I'm no mathematician, but I don't see how these problem types are equivalent. Could you elaborate?
For a type theory we might want to reason about, there’s a diagram (in category theory) which represents the same semantic content. These diagrams turn out to have recurring and common structures.
You can represent those diagrams as adjacency matrices, where those structures have a particular “shape” in the entries. Which if you squint hard looks like an image completion problem, ie, finding missing part of the matrix which represents a proof.
I hadn't heard about it before, although I have some notions of both category theory and theorem proving.
And is your description of “complete and label the diagram” using image generation a thing that actually exists or something that has potential to be created? That could be a breakthrough in applying formal methods to real-world problems.
> And is your description of “complete and label the diagram” using image generation a thing that actually exists or something that has potential to be created?
Somewhere between — it’s a topic being researched, but results are very early (basically, just shapes and groups).
If you think of AI as a dial from sample(data) to mean(data), then as the dial is turned towards the mean() you get more "generic" results, but also more garbled ones.
Copilot is more like a search engine, having turned the dial more towards sample().
The real invention of the NN is simply to provide that dial in a trainable way.
The only change to the "state of the art" is the size of the weights, and how long they take to train. This "advancement" is no more impressive than google indexing more webpages.
There has been no step-change advancement in AI in, perhaps, 50 years. All we see today is a product of hardware, in GPU/CPUs able to compress TBs of data into c. 300GB of weights. And likewise, the internet to provide it and SSDs to hold it.
The "magic" of AI is no more the magic of wikipida, here: copilot is good only because million+ programmers made github good.
It's still little more than a fancy search.
> It's still little more than a fancy search.
I feel like the goalposts have been moved between your two comments. CoPilot is obviously not producing garbled nonsense, and it's also not just printing the top result from StackOverflow. It is producing code that references my variables, does the right thing 50% of the time, and usually compiles.
One of the nice little things is error messages- when I type `if (!foo) { throw ... ` CoPilot is able to complete a nicely formatted and descriptive error message from its understanding of my code. It's not garbled nonsense, and it's not just a search engine.
Does AI deserve the hype it sometimes gets? Not yet. But I think you're going to have to start digging a little deeper for your commentary.
Imagine if we were all self-replicating computers, and certain members of this silicon race began experimenting with making creatures with carbon macro-molecules to create organic intelligence, you could make the same claim in the other direction:
"There has been no step-change advancement in Organic Intelligence in, perhaps, 50 years. All we see today is a product of cell count, in neurotransmitter chemistry able to compress TBs of experiences into c. 300B neurons."
But I agree with you that everything they do seems intelligent because 'intelligence' was in the training data. Not much different from us, if you raise a human removed from society (take his intelligent training data away) he will not accomplish almost anything on his own.
When you are in its bounds it seems magical, once you go outside it seems like a weak joke.
And many of the reasons it is bad outside its sweet spot are fundamental to how it works not a flaw that can be iterated away.
Can you elaborate?
It has a strong sense of what 1 object is. Ask it for a photo of a kitten and you get an jaw droppingly realistic photo of a kitten.
Ask it for a picture of 6 kittens and you get, well, whatever this is: https://labs.openai.com/s/PIZJe6GCfat9soN3WOV3eC9p
7 kittens, none well defined. Because the loss function it is optimising taps out once it has got "close" and close for a multiple object prompt is _a lot_ further away than for a single subject prompt (here is the 1 kitten version https://labs.openai.com/s/1aCOUxNT19kbMZZtEBG7CoFY - this is basically witchcraft it is so good, the group shot is a joke).
I suppose you could massively reduce the loss amount you are willing to accept but that doesn't guarantee dalle with optimise the correct part of the pictures - maybe I'd have just ended up with really, really good floors.
The other thing Dalle is bad at is backgrounds, and once again this is due to "optimising an error score". https://labs.openai.com/s/U1Vo2fxThuXmQZzIwLQ4g9Ai nothing about this is right. At a superficial glance it looks like the view over a city but it's a random splatching of building cutouts and when you look at the detail of the builds they are a blur of pixels that kind of approximate doors and windows but are nothing of the sort. They are super fuzzy and dream like. Because it's trying to generate an image that looks like a cityscape from its memory of Glasgow cityscapes. THere's no coherance because it's trying to covert random pixels into a cityscape not for buildings out of components that humans know go to makeup buildings.
Recall that it's stored a compressed version of TBs of images.
Now, your examples simply show some slight artifacts and lack of details on specific things. You're presenting remaining shortcomings on these metrics as "fundamental to how it works not a flaw that can be iterated away", when in fact they have mostly been iterated away over the past few years.
A larger corpus allows it to be more varied but it doesn't fix the fundamental limitations.
People thought it might take centuries for a computer to defeat a top human in Go. Then deep learning showed up and a few years later it's the opposite.
A lot of the things deep learning methods are doing now are things no one had any idea how long research would take to achieve, or if they were even possible.
Personally, I think we are currently hitting some walls that might take a while to climb before we get to AGI, but I am very impressed at the recent progress.
I'm hoping it's not as bad as Dall-E and Stable Diffusion - I've tried to use those to generate some generic product looking stock photos for a demo and it's spectacularly bad. The only context I see it get praised is fantasy style art - and that is visually appealing nonsense by definition.
If the code generated by copilot has the same "looks convincing but is fundamentally flawed" quality then it sounds like an insidious bug generator.
You'd have to be unrealistically pessimistic to call what GPT-3 and other huge language models produce "nonsense".
AlphaGo was built on that - the model can propose moves, but you can verify who won in the end. There are some code generation models that write their own tests as well, or use externally provided tests to verify their solutions. The DeepMind matrix multiplication algorithm was also "learning from verification" of generated solutions, because it's trivial to do that. In general verification remains an open problem.
Brains were, however, built for language processing, in addition to many other tasks.
> There is no reason to think language models are unfit for math, they are fit for generating possible solutions that need to be verified somehow.
This is just a dumb idea though. Guess and check based on semantically well positioned answers in the ambiguity that is the embedding space until you find something that's not wrong is not the same thing as defining an algorithm and then executing it, which is how people do math.
Sure, you could probably get it to a pretty good working state, but it seems pretty dumb to me.
> There are some solutions called "Language Model Cascades" that compose language models calls to simulate arbitrary complex reasoning chains including recursion.
If you're creating the reasoning chains yourself, you're arguably doing the hard part for the model and giving credit to the language part. If you're able to do get the model to define the chains, then you've already solved the hard part of the problem and could likely use something very different from language models altogether to greater effect.
True, but that's how "inspiration" works in humans as well: generate stupid ideas until you stumble upon a great one.
That's why I said we only need verification. It's the artist-critic model, we got the artist we need the critic. Sometimes it's easy (in games, code, math) and other times we don't have a good way to verify.
I don't agree with this, but even if it were true, "inspiration" is not how we do math.
> That's why I said we only need verification. It's the artist-critic model, we got the artist we need the critic. Sometimes it's easy (in games, code, math) and other times we don't have a good way to verify.
That's not even how the typical language model works.
Honestly, even as someone generally pretty dismissive of the AI hype, I'm not sure you can go that far. The whole reason we have specific mathematical notation is that human languages often are not super great at dealing with it, and English in particular is pretty abysmal for being both unambiguous and precise (and I'd be surprised if language models didn't end up suffering from biases analogous to how many image recognition AI models have been found to not deal well with a diverse set of human appearances). We don't teach math the same way we teach English, and we certainly don't expect people to be experts at teaching both, so why would we expect an AI model designed for language to be able to do math?
The article talks about abstract math questions, but even arithmetic is hard for language models.
For context, Minerva came out this July. When it was tested on a national math exam, it scored higher than that year's class of graduating high school seniors. [2] A mere eight months (!) earlier, OpenAI had announced [3] they'd trained a language model that solved math word problems almost as well as an average middle-schooler. So even if you believe — rightly or wrongly — that current capabilities aren't very impressive, it's worth remembering that your understanding of current capabilities might not be entirely accurate, even if it's only a few months out of date.
Incidentally, it may be worth looking at some examples of these models' outputs before deciding what they can or can't do. Here's Minerva solving some math problems, for example:
https://dl.airtable.com/.attachmentThumbnails/32f10fc59039c7...
https://dl.airtable.com/.attachmentThumbnails/f6f86e0edd14d1...
https://dl.airtable.com/.attachmentThumbnails/d9bc743dca1015...
https://dl.airtable.com/.attachmentThumbnails/51562ef70c2b14...
https://dl.airtable.com/.attachmentThumbnails/c081906cc4d23b...
I'll admit I find it challenging to interpret these results as "passable garbled nonesense [sic]", though perhaps I'm not being demanding enough. At any rate, when these models go from beating 10-year olds at math to beating 18-year olds at math in the span of 8 months, one does start to wonder how much of the hype is really due to over-interpretation — and what the next 8 months have in store.
========================
[1] https://arxiv.org/pdf/1709.04326.pdf (For comparison, a 3-time Mathematics Olympiad gold-medalist scored 90%.)
[2] This was Poland's National Math Exam. Minvera scored 65%; the 2021 national average was 57%. See https://www.aitracker.org/ and Section 3 of the Minerva paper at https://arxiv.org/pdf/2206.14858.pdf
In it, it depicts the growth of a nascent AI from its attempts at understanding humor. The AI befriends a technician and gets the human to rate its own crafted jokes.
Eventually the AI gets really good at telling jokes, and becomes sentient as a result.
It was a very fun take on AI gaining sentience, highly recommended!
That solution has a compressed representation of half the internet.
NNs are "garabled nonesense" insofar as they try to generalise; insofar as they are search engines, they provide apparent sense by just repeating something in their database (= weights).
Google, likewise, returns joke explanations.
"AI Language Models" are not touted as some general AI that is smart at everything, like a clever person with multiple intellectual skills integrated into one.
AI language models are for modeling language, not for math problem solving, or anything else. People good at language aren't always good at math.
DeepL produces very good, correct translations for "Alice has five more balls than Bob, who has two balls after he gives four to Charlie. How many balls does Alice have?" into numerous languages, even though it doesn't offer a solution.
I have little doubt that an AI system could be trained to translate word problems like this into systems of equations, which could be dumped into a some decades-old CAS to obtain a solution, which the AI could map back into the verbal domain through the identities between the math variables like x and Alice's apples.
"Hey look, that human who is supposedly good at math can't produce a painting of the Grand Canyon in the style of Monet, even if given eight months to do it, and is easily defeated in chess."
It's all just passable garbled nonesense that the reader (goes to lengths) to interept based on their prior knowledge, which is not expressed in the syntax of what these systems output.
In the case of mathematics, we're far less willing to "BS away" the interpretive failures. But if we were equally demanding, likewise, all prose generated by these systems isnt AI "getting" anything either.
Pass a film reel thru' a shredder and an art student would still call it a film. Pass math thru' and a mathematician wont. This says more about our ability and inclination to make sense out of nonesense when in apparent communicative situations (since, when speaking to a person, this actually improves our mutual understanding).
So, how much of AI is just hacking people's cognitive failures: (1) people's willingness to attribute intention; (2) people's willingness to impart sense "at all costs" to apparent communication; and (3) "hopeium".