kastnerkyle
I would say in the neural network literature at large, and in audio modeling particularly, this continual back and forth of pushing DSP-based knowledge into neural nets, on the architecture side or data side, versus going "raw-er" to force models to learn their own versions of DSP-style transforms has been and will continue to be a see-saw, as we try to find what works best, driven by performance on benchmarks with certain goals in mind.
These types of push-pull movements also dominate computer vision (where many of the "correct" DSP approaches fell away to less-rigid, learned proxies), and language modeling (tokenization is hardly "raw", and byte based approaches to-date lag behind smart tokenization strategies), and I think every field which approaches learning from data will have various swings over time.
CCD bitstreams are also not "raw", so people will continue to push down in representation while making bigger datasets and models, and the rollercoaster will continue.
[0] https://github.com/acids-ircam/RAVE?tab=readme-ov-file
Also, I think artistic uses (such as Dadabots, who heavily used SampleRNN) show clearly that "musicians" like interesting tools, even if uncontrolled in some cases. Tools to exactly execute an idea are important (DAW-like), but so are novelty generating machines like (many) unconditional generators end up being. Jukebox is another nice example of this.
On the "good for elevator music" comment - the stuff I've heard from these models is rarely relaxing enough to be in any elevator I would ride. But there are snippets of inspiration in there for sure.
Generally, I do favor controllable models with lots of input knobs and conditioning for direct use, but there's space for many different approaches in pushing the research forward.
Different creators will work all kind of odd models into their workflows, even things that are objectively less "high quality", and not really controllable. To me, that's a great thing and reason enough to keep pushing unsupervised learning forward.
I think there are a lot of methods from these older Markovian setups that can be employed in the outputs samplers of modern models, as well as the inclusion of structured searches and so on. Parts of deep learning have always focused on structured output search, but historically the LLM style generative setting has not employed these approaches (though I find beam search for generative settings needs tweaking, it usually works pretty well in smaller scale problems for me).
[0] Avoiding Plagiarism in Markov Sequence Generation, Papadopoulos et. al. https://axon.cs.byu.edu/Dan/673/papers/papadopoulos.pdf
Would be interesting if you could prompt, LoRA distill, or use modern LLM tricks against a well-labeled and curated set, similar to how other tagging problems are handled with modern pretrained models.
[0] https://open.nytimes.com/our-tagged-ingredients-data-is-now-...
(Note: There are many, many great optimization papers since 2014 - I just don't see them show up in general recipes in open source too often)
Would much prefer to see early work with solid small scale results on arXiV, than have people hold concepts for another 6 months scaling up. Let that be for a v2, if you cannot put early but concrete results on arXiV where else is there?
Recalling that a lot of nice papers are mostly MNIST / CIFAR-10 level results at first, followed by scale (thinking of VQ-VAE, PixelCNN / RNN, PerceiverAR, many others that worked well at scale later). That doesn't mean every result will scale up, but we have a lot of tricks to scale "small-scale" models using pretrained latent spaces and so on. The first diffusion results were also pretty small scale... different time but I don't think things are so different today.
That said, I can agree that you need to be a bit in the weeds on the research side to be diving deep on this - but I expect lots of followup clarifications or blog posts on this type of work.
From the descriptions here it sounds a lot like AudioLM / SPEAR TTS / some of Meta's recent multilingual TTS approaches, although those models are not open source, sounds like PlayHT's approach is in a similar spirit. The discussion of "mel tokens" is closer to what I would call the classic TTS pipeline in many ways... PlayHT has generally been kind of closed about what they used, would be interesting to know more.
If you are interested in some recent open to sample-from work pushing on this kind of random expressiveness (sometimes at the expense of typical "quality" in terms of TTS), Bark is pretty interesting [1]. Though the audio quality suffers a bit from how they realize sequences -> waveforms, the prosody and timing is really interesting.
I assume the key factor here is high quality, emotive audio with good data cleaning processes. Probably not even a lot of data, at least in the scale of "a lot" in speech, e.g. ASR (millions of hours) or TTS (hundreds to thousands). As opposed to some radically new architectural piece never before seen in the literature, there are lots of really nice tools for emotive and expressive TTS buried in recent years of publications.
Tacotron 2 is perfectly capable of this type of stuff as well, as shown by Dessa [1] a few years ago (this writeup is a nice intro to TTS concepts). With the limit largely being, at some point you haven't heard certain phonetic sounds before in a voice, and need to do something to get plausible outcomes for new voices.
[0] Discussion here https://github.com/neonbjb/tortoise-tts/issues/182#issuecomm...
[1] https://www.tiktok.com/@jonathanflyfly/video/722513498370947...
[1a] Bark github https://github.com/suno-ai/bark
[2] https://medium.com/dessa-news/realtalk-how-it-works-94c1afda...
There's some neural / patch blends from 2016 that I always thought were interesting (CNN-MRF) [1], and I think there's a renaissance in those approaches recently (combined with other generators / prompts etc.). You can also argue ViT is "patch based" in a major sense... I am still a big believer in patch + combinations + warping (non-parameteric synthesis) generally, some cool older work from Apple on that in speech land [2].
I go as far as arguing BPE / wordpiece / sentencepiece / tokenizers in general are key for modern approaches (as were word vocab selections in the earlier days of NMT), because they find 'good enough' patches (tokens) for a higher level model to stitch together while still having some creativity / generalization available... but we focus on the model details rather than the importance of the tokenizer (and tokenizer distribution) in publication many times.
[0] http://people.eecs.berkeley.edu/~efros/research/quilting.htm...
I've also been reading your paper on these topics, based on previous comments on HN. So thanks for that! Always interested in work about constrained generation.
I'd really like to see some version of the techniques from this palindrome generation https://www.ijcai.org/Proceedings/15/Papers/353.pdf revisited in modern contexts. Or some of the other Sony CSL work on this topic (e.g. https://www.francoispachet.fr/wp-content/uploads/2021/01/pap...) which I will admit I (still) do not understand yet.
Some early personal experiments with adding "prefix-style" context by a cross-attention (in the vein of PerceiverAR) seemed like it really helped things along, which would kind of point to search-like behavior as well.
Probably the closest theory I can think of is orderless NADE, which builds on the "all orders" training of https://arxiv.org/abs/1310.1757 , which in my opinion closely relates to BERT and all kinds of other masked language work. There's a lot of other NAR language work I'm skipping here that may be more relevant...
On discrete diffusion:
Continuous diffusion for categorical data shows some promise "walking the boundary" between discrete and continuous diffusion https://arxiv.org/abs/2211.15089 , personally like this direction a lot.
If you have a pre-made embedding space, SSD-LM is a straightforward method https://arxiv.org/abs/2210.17432
SUNDAE worked well for translation https://arxiv.org/abs/2112.06749 and many other tasks.
My own contribution, SUNMASK, worked reasonably well for symbolic music/small datasets (https://openreview.net/forum?id=GIZlheqznkT), but really struggled with anything text or moderately large vocabulary, maybe due to training/compute/arch issues. Personally think large vocabulary discrete diffusion (thinking of the huge vocabs in modern universal LM work) will continue to be a challenge.
Decoding strategies:
As a general aside, I still don't understand how many of the large generative tools aren't exposing more decoding strategies, or hooks to implement them. Beam search with stochastic/diverse group objectives, per-step temperature/top-k/top-p, hooks for things like COLD decoding https://arxiv.org/abs/2202.11705, minimum Bayes risk https://medium.com/mlearning-ai/mbr-decoding-get-better-resu..., check/correct systems during decode based on simple domain rules and previous outputs, etc.
These kinds of decoding tools have always been a huge boost to model performance for me, and having access to add in these hooks to "big API models" would be really nice... though I guess you would need to limit/lock compute use since a full backtracking search would pretty swiftly crash most systems. Maybe the new "plugins" access from OpenAI will allow some of this.
As an aside - do you have any academic references for masking / constraining based next step transitions? It's a trick I've been messing with for years, but I probably picked this up from some older poetry generation work. Haven't been able to find any direct reference for simple 1 step masking, let alone more involved things like constraining based on the unrolled, modified markov sequence or things bigger than 1 step.
See the images in this stack overflow for an example [0].
But a lot of the diffusion literature does focus the math, since finding tighter bounds and proving that things converge to the true likelihood etc. etc. are current and recent contributions in research (cf https://proceedings.neurips.cc/paper/2021/hash/c11abfd29e4d9... or https://proceedings.neurips.cc/paper/2021/hash/b578f2a52a022...)
The summaries by Yang Song (https://yang-song.github.io/blog/2021/score/) and Lilian Weng (https://lilianweng.github.io/posts/2021-07-11-diffusion-mode...) are arguably the definitive summaries, but there is math there too.
Personally the idea of training a model that is learning to go from (more noise -> less noise) stepwise is a pretty intuitive one (used to call it iterative inference I guess), but that simple message does get wrapped up in proofs and theorems quite a lot in the literature right now.
If you make analogy to GAN generators, which go from noise -> data in one shot (and presumably might need to do this kind of iteration/denoising, implicitly and internally), you are kind of relaxing the modeling problem and allowing for variable compute time at prediction (as opposed to trying to train a GAN with a huge number of layers in the generator).
Similar analogies also hold when looking at the VAE formulation, and seeing it as a mapping from gaussian noise (latent/Z) to data via the decoder following in the tradition of latent variable modeling setups like LDA, with the encoder being a practical and useful necessity to map into this latent space (some early slides from Durk Kingma and Max Welling present/relate VAE in this light - particularly the "plate diagram" representation of VAE highlights this well). Similar analogies also hold for flow based models, and are used frequently to define and teach about flow-based generative models (https://lilianweng.github.io/posts/2018-10-13-flow-models/).
Ultimately (in my opinion) each of these branches has their own "math corner" people spend time in - minmax game stuff for GAN, ELBO / bounds for VAE (or deriving new priors), bijection / invertibility in flows, and now noise schedules for diffusion. Just part of research I guess.
But these diffusion models are pretty straightforward to train, and pretty powerful in my experience so far - definitely worth cutting through the noise if you are interested in generative models but (like me) aren't overly invested in the math-parts
Jascha's slides with the "dye in water" analogy (starting ~slide 18, https://www.lri.fr/TAU_seminars/videos/Jascha_Sohl_Dickstein...) are a great intuitive introduction to the concept
For anyone unfamiliar with diffusion models (and coconet / OrderlessNADE), one of the really nice properties of them as opposed to "standard" autoregressive (GPT / RNN) style models, is that you should be able to specify any part, and fill in any other part - rather than being forced to specify the "past" and predict only the "future". The coconet "doodle" is a good example of this interface at work (https://www.google.com/doodles/celebrating-johann-sebastian-...)
XLNet had some of this promise too (https://arxiv.org/abs/1906.08237) but I never had much luck with it as a pure generator. Autoregressive Diffusion models (https://openreview.net/forum?id=Lm8T39vLDTE) have similar properties, but I haven't had time to sus out the subtle differences yet.
You can roll-back the generation process and/or mask the probability distribution using simple secondary logic, but I find beam search gives generally better results, especially when the word I want to force is very low probability - most of my sequence models kind of go off the rails when they are forced into a low-probability sequence ("the man went to the xylophone zebra sdawoqhdjwna"). Also I find this problem gets worse in domains without "reset" tokens like spaces, where there are always high entoropy possibilities (the letter after a space has a lot of good choices) followed by lower ones (after the first letter, there often become less good choices - at least until you hit another space). Particularly in music generation, models that sample a "surprising" sequence tend to go off the rails. It is also a behavior that seems worse in RNNs, than transformers for me.
Some of the better ones I used were variants of diverse beam search, and stochastic beam searches usually combined together. The "classic" / pure variant has generally not been as useful in generative modeling for me, it tends to collapse to basically one or two effective candidates (with maybe some filler words changed) fairly quickly.
Also it seems to generally work better for me in conditional generation than in unconditional generation (e.g. charRNN / some uses of GPT-2). However, things like the "repetition problem" can be removed by construction if you are willing to hack in the beam search just a little bit. See https://badsamples.tumblr.com/post/160777871547/stochastic-s... (stochastic, diverse beam search w Markov iirc) vs https://badsamples.tumblr.com/post/160767248407/a-markov-arg... (fixed beam search, where I didn't try to remove repetition or anything special, same Markov setup)
Sometimes I also manipulate probabilities with masks and things directly, and that also combines fine with beam search in the experiments I have done.
Nucleus sampling works well, and if you don't want to control or constrain the output in unconditional generation I don't know that beam search really does much. But for conditional generation, or post-hoc hacks to get more control over a generator I find beam search variants really useful. Especially combined with a specifically conditional architecture.
For example, conditioning the language model on a particular bag-of-rhyme-words + (stochastic, probably) beam search to force rhyme pairs at the start and end of lines, probably further modified by input and output masks to "blank out" invalid tokens and tell the model which tokens will be blanked out. I've used some blend of these tricks in speech, music, and text experiments and it can be helpful if you have structure that is important to replicate and a simple model, with simple sampling just isn't replicating the necessary structure.
EDIT: One practical reason to do this would be plagiarism detection, especially if fine-tuning a small corpus. There are ways with guarantees by construction (https://www.researchgate.net/profile/Pierre_Roy2/publication...) but simple setups using beam searches and tries can also do constraint checks for n-grams of certain lengths. Concretely, set up tries for 1-2-3-4-...-nminus1-grams, which are considered "valid" transitions, then set a "bad" trie for n-grams. Check these tries during generation, and throw out any candidates which violate the "bad" trie, but still match in the good one.
See the line of Max Order work from the Sony CSL lab (formerly run by Francois Pachet) for some examples of this.
Might be worth investigating if you are interested in this application.
This all gets worse if you want multi-speaker output of course - getting even a few speakers who all read the same large corpus is difficult. The two datasets I've gotten the most out of so far are "LJSpeech" (a subset of the LibriVox corpus), and the "Nancy Corpus / Blizzard 2013" dataset [0][1].
There's a pretty interesting corpus here that I hope to start using soon [2].
To me, the biggest issue / gap between commercial interests and publically available data is curation - TTS really hinges on well curated, clean data at least for now. And if that dataset has a very balanced coverage of triphones, that's even better.
I'd like to try on the voice.mozilla data, but given current stuggles on even 1 speaker, a truly "in the wild" set of many speakers seems pretty difficult if training from scratch. For voice cloning using pretrained weights it may be a different story.
[0] https://keithito.com/LJ-Speech-Dataset/
[1] https://www.synsig.org/index.php/Blizzard_Challenge_2013
[2] http://www.m-ailabs.bayern/en/the-mailabs-speech-dataset/
The best quality I have heard in OSS is probably [1] from Ryuichi using the Tacotron 2 implementation of Rayhane Mamah, which is loosely what NVidia based some of their baseline code on recently as well [3][4].
There's also a colab notebook for this stuff, so you can try it directly without any pain https://colab.research.google.com/github/r9y9/Colaboratory/b...
I also have my own pipeline for this (using some utilities from the above authors + a lot of my own hacks), for a forthcoming paper release here https://github.com/kastnerkyle/representation_mixing/tree/ma... , see the minimal demo. It has pretty fast sampling, but the audio quality is not as high as WaveNet. I'd really like to tie in with WaveGlow, but it's work in progress for me so far.
NOTE: None of these have voice adaptivity per se, but given a model which trains well already + a multispeaker dataset with IDs such as VCTK, a lot of things become possible as getting a baseline model and data pipeline for TTS is quite difficult.
[0] https://github.com/keithito/tacotron
[1] https://r9y9.github.io/blog/2018/05/20/tacotron2/
[2] https://github.com/Rayhane-mamah/Tacotron-2
We followed this approach in char2wav [2], but "voice cloning" has come much farther in my opinion [3][4][5]. There's a lot of relevant research on techniques for this beyond concatenating indicators or embeddings, if people are interested in the research side of this technology.
[0] https://arxiv.org/abs/1308.0850
[1] https://www.youtube.com/watch?v=-yX1SYeDHbg&t=38m30s
[2] http://josesotelo.com/speechsynthesis/
[3] https://twitter.com/Jeanne_Heo/status/972089715225542657 (lyrebird.ai)
[4] https://google.github.io/tacotron/publications/gmvae_control...
[5] https://google.github.io/tacotron/publications/speaker_adapt...
If you like this, you might like some more recent work from Anna huang et. al. at Magenta, using a modified Transformer network [2][3].
Some other relevant reading includes DeepBach [4][5], and outside the domain of Bach alone, the venerable folkRNN [6]. There's a huge chain of work I posted a while back here [7], and Kyle McDonald wrote a really great review piece here [8].
Modeling music in latent spaces has also seen a number of recent publications [9][10][11], and having smooth latent spaces for generation seems (to me) to be a very natural interface for manipulating things, which has a natural analogue in "real" composition, cool video at [12].
There have also been some really fun online demos for interacting with these types of systems [13]
There's a lot of interesting work going on in music modeling, hopefully leading to new and interesting musical tools and toys. Interactivity (often via clever conditioning) can really change the way you approach these types of problems [14].
[0] https://github.com/kastnerkyle/magenta/tree/my_cleanup_branc...
[1] https://www.youtube.com/watch?v=Ay8mHW0-NW8&list=PLRMa_gJ8vx...
[2] https://twitter.com/kastnerkyle/status/1018617315196309504
[3] https://arxiv.org/abs/1809.04281
[4] https://www.flow-machines.com/archives/deepbach-polyphonic-m...
[5] https://github.com/Ghadjeres/DeepBach
[6] https://github.com/IraKorshunova/folk-rnn
[7] https://news.ycombinator.com/item?id=14822281
[8] https://medium.com/artists-and-machine-intelligence/neural-n...
[9] https://storage.googleapis.com/magentadata/papers/multitrack...
[10] https://magenta.tensorflow.org/music-vae
[11] https://www.youtube.com/watch?v=BbyvbO2F7ug
[12] https://www.youtube.com/watch?v=A8d-PUuUJnA
[13] https://codepen.io/teropa/
[14] https://twitter.com/chrisdonahuey/status/1051868153700016128
[0] https://link.springer.com/article/10.1007/s10601-010-9101-4
""" Prompt: The Mage, the Warrior, and the Priest
Story: A light breeze swept the ground, and carried with it still the distant scents of dust and time-worn stone. The Warrior led the way, heaving her mass of armour and muscle over the uneven terrain. She soon crested the last of the low embankments, which still bore the unmistakable fingerprints of haste and fear. She lifted herself up onto the top the rise, and looked out at the scene before her. [...] """
RNN/LSTM type models can also produce really coherent results on the scales shown in the blog.
Markov chains can be quite useful when you can manually create conditioning, but some of the key power of neural methods is learning to process conditioning and blend in ways that may not be obvious (or require significant domain expertise to grok) when handcrafting. YeGoblynQueenne below discusses a lot of the core issues for Markov chain based approaches.
In particular, plagiaristic sequences are incredibly common in Markovian generation, and though there are some papers on how to deal with this [1] it is not straightforward to decide what plagiarism really means in many contexts. This problem also arises in neural models, but isn't nearly as extreme due to the nature of both learning and the sampling process.
One huge bonus of Markov chains is that controlling exactly what you want or using hard rules is pretty easy, which is definitely NOT the case with neural generation...
This blog post is a really nice run through of how to get decent results from Markov chains, but don't discount neural methods either - they are getting better all the time, if you are willing to deal with the headaches.
[0] Hierarchical Neural Story Generation, https://arxiv.org/abs/1805.04833
[1] Max Order http://www.flow-machines.com/maxorder/
Followups to soundex such as metaphone are encumbered by license issues as far as I know, but Caverphone is free and clear AFAIK.
[2] is an insanely great overview of many of these algorithms, be sure to check it out if you are into this stuff.
[0] https://caversham.otago.ac.nz/files/working/ctp150804.pdf
[1] https://gist.github.com/kastnerkyle/a697d4e762fa8f53c70eea7b...
[2] http://ntz-develop.blogspot.ca/2011/03/phonetic-algorithms.h...
I found LTSD pretty robust compared to simpler energy based things as long as you have a small chunk of background sound at the start. The LTSD implementation is largely from my friend Joao, so I can't take credit for the cool part, only the bugs
[0] https://gist.github.com/kastnerkyle/a3661d6be10a0ae9e01fd429...