News for hackers. Stuff that matters.
Re:SoundStream: An End-to-End Neural Audio Codec by esjeon on Sunday August 15, 2021 @06:31AM (#28187677)
The post is contaminated with some wet marketing language, adding unnecessary noise to the information. The most important part of this whole post is this:

... [V]ector quantization ... works well at bitrates around 1 kbps or lower, but quickly reaches its limits when using higher bitrates. For example, even at a bitrate as low as 3 kbps, and assuming the encoder produces 100 vectors per second, one would need to store a codebook with more than 1 billion vectors, which is infeasible in practice.

In SoundStream, we address this issue by proposing a new residual vector quantizer (RVQ), consisting of several layers ...

Re:SoundStream: An End-to-End Neural Audio Codec by michaelmdresser on Sunday August 15, 2021 @10:22AM (#28188962)
What is “wet marketing language?”
Re:SoundStream: An End-to-End Neural Audio Codec by throwaway81523 on Sunday August 15, 2021 @12:04AM (#28186178)
This is really impressive. They say that at 3kbps this new codec sounds as good as Opus at 12kbps, and that it is trained across a wide range of bandwidth. It is a much bigger deal than Lyra was. I'd like to know whether it can run at low latency.
Re:SoundStream: An End-to-End Neural Audio Codec by mdp2021 on Sunday August 15, 2021 @06:26AM (#28187664)
> They say that at 3kbps this new codec sounds as good as Opus at 12kbps

With that jump in quality, makes you wonder about the processing power consumption ratio.

Re:SoundStream: An End-to-End Neural Audio Codec by amelius on Sunday August 15, 2021 @04:58AM (#28187261)
Low latency was a design goal.
Re:SoundStream: An End-to-End Neural Audio Codec by mchusma on Sunday August 15, 2021 @12:17PM (#28189968)
How low though? Lyra added significant overhead, from what I read. They said it was small but then in practice it was like 100ms or something, which given a typical ok connection is easily 50ms would put it in the perceivable range of delay.
Re:SoundStream: An End-to-End Neural Audio Codec by taneq on Sunday August 15, 2021 @12:19AM (#28186236)
Any sufficiently fancy compression for communication formats immediately makes me worry about the Xerox Effect[1], where the reconstructed content is valid yet semantically different in some important way to the original.

[1] I propose we call it that unless there's already a snappy name for it?

Re:SoundStream: An End-to-End Neural Audio Codec by oddity on Sunday August 15, 2021 @01:19AM (#28186437)
Indeed. I also expect this failure mode will be undetected for a long time due to how our sense of hearing works. My last neuroscience class was many years ago, but I do remember that in some sense, we hear what we expect to hear (more so than vision if I recall correctly, though there is plenty that happens in our vision processing) in that our ears tune for particular frequencies to filter out ambiguities.

Suppose a person says something that the codec interprets differently. Perhaps they have one of the many ever evolving accents that almost certainly were not and absolutely could not possibly be included in the training set (ongoing vowel shifts might be a big cause of this). The algorithm removes the ambiguity, but the listener can't tell because they hear themselves through their own sense of hearing. Assume the user has somehow overcome the odd psychological effects that come hearing the computer generated audio played back, if that audio is mixed with what the person is already hearing, it's likely they still won't notice because they still hear themselves. They would have to listen to a recording some time later and detect that the recording doesn't match up with what they thought they said... which happens all the time because memory is incredibly lossy and maleable.

Most of the time, it won't matter. People have context (assuming they're actually listening, which is a whole other tier of hearing what you expect to hear) and people pick the wrong word or pronounce things incorrectly (as in not recognizable to the listener as what the speaker intended) all the time. But it'll be really hard to know that the recording doesn't record what was actually said. You need to record a local accurate copy, the receiver's processed copy, and know to look for it in what will likely be many hours of audio. It's also possible that "the algorithm said that" will be a common enough argument due to other factors (incorrect memory and increasing awareness of ML-based algorithms) that it'll out number the cases where it really happens.

Re:SoundStream: An End-to-End Neural Audio Codec by skybrian on Sunday August 15, 2021 @02:33AM (#28186661)
This seems similar to being able to read your own handwriting, when others can't. If it's an important recording, someone else should listen, and it would be better to verify a transcription.

In a live situation, they will ask you to repeat if it's unclear.

Re:SoundStream: An End-to-End Neural Audio Codec by spyder on Sunday August 15, 2021 @05:07AM (#28187311)
Yep, it's kinda happening with the music example on the page: the Lyra (3kbps) sample have some human sounding part when the original reference is just music without any speech. Probably because Lyra was trained on speech.
Re:SoundStream: An End-to-End Neural Audio Codec by im3w1l on Sunday August 15, 2021 @05:02AM (#28187283)
It's a valid concern but I think it can also be solved. Compress-decompress and compare to the original using a method not susceptible to xerox effect. If the sound has materially changed then use a fallback method, robust but less efficient, for that particular window.

But idk this may be too slow for real time.

Re:SoundStream: An End-to-End Neural Audio Codec by userbinator on Sunday August 15, 2021 @01:00AM (#28186384)
I remember that: https://news.ycombinator.com/item?id=6156238

I agree, neural networks are exactly the type of system that works well "most of the time" but then can fail unexpectedly in odd and subtle ways.

Re:SoundStream: An End-to-End Neural Audio Codec by spindle on Sunday August 15, 2021 @02:58AM (#28186757)
Re:SoundStream: An End-to-End Neural Audio Codec by thrdbndndn on Sunday August 15, 2021 @12:01AM (#28186160)
Just curious, is there any progression for relatively high bitrate audio codec? Not that I'm not satisfied with the current state of AAC, but I found most of these new development often about some super low bitrate (this case is even more extreme, 3kbps!?).
Re:SoundStream: An End-to-End Neural Audio Codec by chrismorgan on Sunday August 15, 2021 @12:20AM (#28186242)
Opus.

Opus is a remarkable codec because it’s excellent at almost everything. The only areas where it’s being beaten are extreme narrowband, which it can’t do, and narrowband, where it’s still not shabby (though some of this new stuff is redefining what’s possible).

Opus tackled a broad field of competitors that were each somewhat specialised for their part of the field, and pretty much beat all of them at their own game. And in most cases the incumbents were high-latency, while Opus not only achieves quality superiority but also supports low-latency operation.

https://en.wikipedia.org/wiki/Opus_(audio_format) has some good diagrams and explanation. https://upload.wikimedia.org/wikipedia/commons/8/8d/Opus_qua... especially shows Opus’ superiority except in narrowband.

Past about 16kb/s, Opus is pretty much just the format to use, except for extreme niches like if you want to represent over 20kHz (above which Opus cuts).

Opus is so good there’s pretty much nothing left to improve for now (for now), and even if you improved things it probably wouldn’t be worth it. That’s why all the development is happening in narrowband, because that’s the only interesting space left. Perhaps eventually some of those techniques will be scaled up past narrowband. I don’t know.

Re:SoundStream: An End-to-End Neural Audio Codec by makomk on Sunday August 15, 2021 @04:58AM (#28187259)
Yeah. In reality the main reason new audio codecs are developed post-Opus isn't technical, it's so that companies can get their patents into new standards and rake in the licensing royalties. There are better codecs for really low bitrates but that's quite niche these days; even telephony is going wideband and higher bitrate.
Re:SoundStream: An End-to-End Neural Audio Codec by sneak on Sunday August 15, 2021 @06:03AM (#28187571)
I think you perhaps overestimate the bandwidth available to most telephone handsets in most of the world.

I'm in a hotel in a schengen country right now and I am lucky to get 200kbps via the wifi.

Other, less industrialized countries are frequently worse.

It's safe to say a ~billion people's lives can be improved with better low bitrate codecs.

Re:SoundStream: An End-to-End Neural Audio Codec by tialaramex on Sunday August 15, 2021 @09:21AM (#28188568)
Let's assume for a moment that you're not stupid enough to confuse 200kBps (1.6Mbps) for 200kbps.

Opus is fine down to 8kbps. It fits over a cheap, shitty mid-20th century analogue telephone line with room to spare.

The ultra-narrow band stuff is very niche, and is consequently unlikely to have the broad impact you're imagining.

In contrast there is continue enthusiasm for these pointless midband codecs that are similar in performance to Opus but have the "advantage" that somebody gets $$$.

Re:SoundStream: An End-to-End Neural Audio Codec by zinekeller on Sunday August 15, 2021 @09:32AM (#28188643)
"Pointless" is relative here.

8kbps is enough - when you don't spare anything anything to bit correction. Maybe in those cases, somewhat okay analogue audio is enough (for example, in long-distance raditelephony). But having a very impressive digital codec raises the bar significantly, especially the last time someone bothered with this is someone in Nokia trying to fit 6kbps using what was now rudimentary phone chips.

Additionally, there are people in the world (including US) who are stuck using unreliable 28kbps lines. Having an option to do excellent audio and video is something that no-one seemed bothered to do.

Re:SoundStream: An End-to-End Neural Audio Codec by sneak on Monday August 16, 2021 @03:24PM (#28201796)
That's 200kbps for a building with a few dozen people in it.
Re:SoundStream: An End-to-End Neural Audio Codec by vbezhenar on Sunday August 15, 2021 @10:24AM (#28188978)
Why Whatsapp calls are so terrible even on 100 Mbps links?
Re:SoundStream: An End-to-End Neural Audio Codec by diimdeep on Sunday August 15, 2021 @06:02AM (#28187567)
I just compared 730 kb/s Flac with 160kb/s Opus, I can see no difference even on spectrogram using 'mother of mp3' track: Tom's Diner (Live At The Royal Albert Hall, London, UK, November 18, 1986).

Very surprising, will be migrating all my music to opus to save space.

Re:SoundStream: An End-to-End Neural Audio Codec by jacquesm on Sunday August 15, 2021 @07:15AM (#28187838)
Beware of phase differences, they won't show up on a spectrogram but could seriously upset your stereo impression. Before you compress all your Flac content and only figure this one out afterwards. That could be quite annoying.
Re:SoundStream: An End-to-End Neural Audio Codec by Groxx on Monday August 16, 2021 @02:41AM (#28195531)
Yeah - definitely keep your backups in FLAC. Having a lossless source gives you infinite future flexibility, and that cannot be underestimated. Otherwise you're kinda doomed to hit https://www.youtube.com/watch?v=fZCRYo-0K0c eventually.

For on-listening-device though, oh heck yes - Opus is great.

Re:SoundStream: An End-to-End Neural Audio Codec by rokweom on Sunday August 15, 2021 @06:17AM (#28187632)
Just as a side note, spectrogram are not in any way indicative of compression quality.

Its interesting that you didn't see any differences between flac and opus, as opus has a hard 20 kHz low-pass filter and is 48kHz-only.

Re:SoundStream: An End-to-End Neural Audio Codec by diimdeep on Sunday August 15, 2021 @06:58AM (#28187767)
There is visible 20 kHz LPF, but it is meaningless for use case of music listening, right ?

I did not liked degradation at 140-128kb/s and noise above 16kHz and some artifacts 10-16khz that is clearly visible and much more impactful.

Here is spectrograms https://disk.yandex.com/d/PvQGSS2xBu7ucQ

Re:SoundStream: An End-to-End Neural Audio Codec by rokweom on Sunday August 15, 2021 @09:00AM (#28188435)
As I said, spectrograms are not indicative of compression quality. Codecs should be judged by ears only. What you see in a spectrogram will vastly differ between codecs and will not reflect their compression efficiency.
Re:SoundStream: An End-to-End Neural Audio Codec by Nasreddin_Hodja on Monday August 16, 2021 @05:15AM (#28196271)
> I just compared 730 kb/s Flac with 160kb/s Opus, I can see no difference even on spectrogram

Doesn't Ogg Vorbis perform better than Opus on bitrates above 128kbps?

Re:SoundStream: An End-to-End Neural Audio Codec by NavinF on Tuesday August 17, 2021 @04:40AM (#28207575)
Opus is transparent at 160kb/s, so no.
Re:SoundStream: An End-to-End Neural Audio Codec by Nasreddin_Hodja on Tuesday August 17, 2021 @09:17AM (#28209500)
When comparing, your vorbis examples were encoded with 48kHz too?
Re:SoundStream: An End-to-End Neural Audio Codec by NavinF on Sunday August 15, 2021 @04:29AM (#28187111)
For 99.9% of use cases, that's a solved problem. You'll never use anything other than FLAC and Opus for lossless and lossy compression respectively.

I'm sure there are unusual cases such as live streaming over satellite internet where getting an extra 1% compression on high quality audio is a big deal, but even that's likely a temporary problem. Starlink users already get >50mpbs

Re:SoundStream: An End-to-End Neural Audio Codec by ugjka on Sunday August 15, 2021 @04:45AM (#28187193)
I'm sure Sirius XM would be interested in something that outperforms AAC v2 by a large margin
Re:SoundStream: An End-to-End Neural Audio Codec by CharlesW on Sunday August 15, 2021 @10:11PM (#28194317)
> You'll never use anything other than FLAC and Opus for lossless and lossy compression respectively.

That's true only for if you don't mind losing a substantial portion of your potential audience.

In contrast, AAC (including low-bitrate HE-AAC and HE-AACv2 flavors) is ubiquitous as MP3 for anything made in the past 10 years.

Re:SoundStream: An End-to-End Neural Audio Codec by crazygringo on Sunday August 15, 2021 @12:39PM (#28190177)
AFAIK, not really.

After AAC, researchers realized that AAC produced perceptually transparent audio with excellent compression and that growing bandwidth and disk space meant there wasn't much point to further improvement in that direction.

And remember that, unlike MP3, AAC is more like an entire toolbox of compression technologies that applications/encoders can use for a huge variety of purposes and bitrates -- so if you need better performance, there's a good change you can just tune the AAC settings rather than need a whole new codec.

So research shifted in two other directions -- super low-bandwidth (like this post) for telephony etc., and then spatial audio (like with AirPods).

Re:SoundStream: An End-to-End Neural Audio Codec by ksec on Sunday August 15, 2021 @04:32PM (#28192252)
Oh I wrote something about that [1] on HN and HydrogenAudio But basically there are zero incentive to do so. We are no longer limited by Storage or Bandwidth. Bandwidth or Cost Per transfer decline at a much greater rate than any Audio or Video Codec Advancement.

So if you want higher quality at a relatively high bitrate? Use Higher Bitrate. Just use 256Kbps AAC-LC instead of 128Kbps, all of its patents has expired so it is truly patent free. The only not so good thing is that all the Open Source AAC Encoder are't anywhere near as good as the one iTunes is provided. Or you could use Opus, at a slightly better quality / bitrate if it fit your usage scenario. Even high bitrate MP3 for quite literally 100% backward compatibility. If you want something at 256Kbps+ but doesn't want to go Lossless, Musepack continues to be best in its class but you get very little decoding support.

[1] https://news.ycombinator.com/item?id=10787455

Re:SoundStream: An End-to-End Neural Audio Codec by montag on Sunday August 15, 2021 @12:47AM (#28186339)
The article mentions that it can scale bitrate directly by adding or removing layers. But I sure wish they had included some hi-fi quality sample audio.
Re:SoundStream: An End-to-End Neural Audio Codec by enduku on Sunday August 15, 2021 @12:06AM (#28186188)
General Harmonics[0] may have come up with something. [0]www.generalharmonics.com
Re:SoundStream: An End-to-End Neural Audio Codec by amelius on Sunday August 15, 2021 @05:06AM (#28187301)
Would anyone compress their music library using a deep learning codec, given the possibility of awkward artifacts?
Re:SoundStream: An End-to-End Neural Audio Codec by userbinator on Sunday August 15, 2021 @01:20AM (#28186445)
I wonder how big the codec itself is. Neural networks are not exactly small. Also, the processing latency - Lyra mentions on its GitHub page 100ms.
Re:SoundStream: An End-to-End Neural Audio Codec by sanxiyn on Sunday August 15, 2021 @01:36AM (#28186493)
The paper says 8.4 million parameters for the default model. Assuming 4 bytes per parameter, that's about 30 MB.
Re:SoundStream: An End-to-End Neural Audio Codec by Scoundreller on Sunday August 15, 2021 @02:39AM (#28186681)
Being able to download in advance is huge. Looking forward to satellite audio for everything spoken-voice one day. Some org was working on it, but they were working with 2kbps 24/7, which is definitely not a lot!
Re:SoundStream: An End-to-End Neural Audio Codec by benob on Sunday August 15, 2021 @02:17AM (#28186618)
In the papier they compare lattencies from 7ms to 26ms and observe no loss of quality. Processing requirement is lower for higher latencies because of batching.
Re:SoundStream: An End-to-End Neural Audio Codec by ngrilly on Sunday August 15, 2021 @03:53AM (#28186959)
Next step: train a NN to predict what we’re going to say to compensate network latency :)
Re:SoundStream: An End-to-End Neural Audio Codec by vletal on Sunday August 15, 2021 @04:01AM (#28186986)
Have you ever played a counterstrike with a lagging network? Observe the teammates running into walls and such. That's what the conversation would sound like.
Re:SoundStream: An End-to-End Neural Audio Codec by vbezhenar on Sunday August 15, 2021 @10:26AM (#28188994)
Running into a wall is not a result of NN prediction. It's a result of naïve linear prediction. NN will predict that he'll stop, turn around, may be shot someone. The hard thing is to not disappoint NN by actually running into a wall.
Re:SoundStream: An End-to-End Neural Audio Codec by vletal on Sunday August 15, 2021 @02:49PM (#28191314)
You need to take into consideration the complexity. Counterstrike has very tiny set of possible actions you can take. Compare that to the space of things one can say.

A NN is going to fail on that at least as missereably as linear interpolation fails in CS.

Re:SoundStream: An End-to-End Neural Audio Codec by hypertele-Xii on Monday August 16, 2021 @06:31AM (#28196617)
When speech prediction fails, it should sound clearly wrong. Otherwise we risk serious misunderstanding when the prediction says something that sounds good but means different.

We already have trouble like this with texting autocorrection.

Re:SoundStream: An End-to-End Neural Audio Codec by meowface on Sunday August 15, 2021 @06:47AM (#28187719)
It could make for a hilarious new comedic art form.
Re:SoundStream: An End-to-End Neural Audio Codec by lxgr on Sunday August 15, 2021 @04:18AM (#28187064)
Don‘t modern codecs already do something like this to conceal packet loss? (Obviously only at the phoneme and not word or semantic level.)
Re:SoundStream: An End-to-End Neural Audio Codec by amelius on Sunday August 15, 2021 @05:01AM (#28187275)
Does this work for the starting phoneme of a word?
Re:SoundStream: An End-to-End Neural Audio Codec by pbhjpbhj on Sunday August 15, 2021 @06:53AM (#28187748)
Just running with that a little ...

So, at what point during a phoneme does it become distinct?

After any one phoneme has been sent down the wire you can truncate the audio and send a phoneme identifier; at the other end they just replay the phoneme from a cache.

Like doing speech to text, but using phonemes, and sending the phoneme hash string instead of the actual audio.

Must have been done already?

Re:SoundStream: An End-to-End Neural Audio Codec by ngrilly on Sunday August 15, 2021 @04:12PM (#28192084)
Wondering the same.
Re:SoundStream: An End-to-End Neural Audio Codec by mchusma on Sunday August 15, 2021 @04:10PM (#28192067)
Googles duo already does this. In my experience, it's amazing. http://ai.googleblog.com/2020/04/improving-audio-quality-in-...
Re:SoundStream: An End-to-End Neural Audio Codec by ngrilly on Saturday August 21, 2021 @11:24AM (#28257722)
I remember reading that paper. This is amazing. But it's not really predicting what we're gone say beyond the current syllable:

> While our model learns how to plausibly continue speech, this is only true on a short scale — it can finish a syllable but does not predict words, per se.

Re:SoundStream: An End-to-End Neural Audio Codec by johntb86 on Sunday August 15, 2021 @03:43AM (#28186928)
Has anyone thought of trying to end-to-end train an h.265 decoder? The results might not be 100% perfect, but the resulting codec might bypass a ton of patents.
Re:SoundStream: An End-to-End Neural Audio Codec by vletal on Sunday August 15, 2021 @03:59AM (#28186981)
Since the h265 relies heavily on operations which are not easily differentiable, such as translation of patches of images, together with a pretty complicated binary format, I'd be pretty amazed if the NN actually learned anything meaningful at all.
Re:SoundStream: An End-to-End Neural Audio Codec by svantana on Sunday August 15, 2021 @06:06AM (#28187585)
Interpolated translation is continuous and easily differentiated. There's lots of work on machine-learned video codecs already, from Nvidia, Qualcomm and others.
Re:SoundStream: An End-to-End Neural Audio Codec by zinekeller on Sunday August 15, 2021 @07:38AM (#28187941)
> There's lots of work on machine-learned video codecs already, from Nvidia, Qualcomm and others.

I've tried them, the current state of the art means that they're only useful on relatively static things (some shaking etc) while spike up to AV1-level bitrate to reach perceptual similarity when the movement is too much. Maybe in the future (or whatever under the wraps concoction Nvidia, Qualcomm or another player have), ML-based video codecs will surpass handtuned codecs, but it's not (yet) the present state.

Re:SoundStream: An End-to-End Neural Audio Codec by vletal on Sunday August 15, 2021 @06:08AM (#28187594)
It's not difficult to propagate gradients while translating an image. Learning "pick a 8x8 patch from (145,17) apply X to it and translate it by (4,-8)" from data is on different level, is not it?

The premise was using e2e learning to avoid patent issues. I am sure that with some preprocessing you can plug a NN inside the deciding process and learn very meaningful stuff.

Re:SoundStream: An End-to-End Neural Audio Codec by itsamesicario on Sunday August 15, 2021 @04:44AM (#28187188)
Aren't software implementations royalty free anyway? and even if they weren't, could they not classify a neural network as a software implementation? (from the patent enforcing point of view) Because if that is not the case, this idea would be applicable to a lot of things right? Seems like an easy hack
Re:SoundStream: An End-to-End Neural Audio Codec by runnerup on Sunday August 15, 2021 @06:48AM (#28187726)
> Because if that is not the case, this idea would be applicable to a lot of things right? Seems like an easy hack

Well that does seem to be Microsoft's position at least. (see: Copilot)

Re:SoundStream: An End-to-End Neural Audio Codec by hliyan on Sunday August 15, 2021 @12:48AM (#28186345)
Is it possible to apply this same technique to video codecs? If not general video, then at least video streams where the center subject is a human face?
Re:SoundStream: An End-to-End Neural Audio Codec by est31 on Sunday August 15, 2021 @12:50AM (#28186355)
Re:SoundStream: An End-to-End Neural Audio Codec by cblconfederate on Sunday August 15, 2021 @08:44AM (#28188330)
> that induce the reconstructed audio to sound like the uncompressed original input.

What if we want the codec to produce easier-perceptible sound, not just a reconstruction. Could bake in noise and echo reduction, voice improvements etc

Re:SoundStream: An End-to-End Neural Audio Codec by vlovich123 on Sunday August 15, 2021 @10:09AM (#28188839)
Did you read the “ Joint Audio Compression and Enhancement” section part of the announcement?
Re:SoundStream: An End-to-End Neural Audio Codec by cblconfederate on Sunday August 15, 2021 @10:57AM (#28189269)
I wonder how the training works in that system
Re:SoundStream: An End-to-End Neural Audio Codec by rcthompson on Sunday August 15, 2021 @01:38PM (#28190617)
I assume think they start with the "signal" and "noise" as separate audio files, and then they play them together in order to create a synthetic noisy input. Then they can train the output against only the signal so that it will learn to filter out the noise.
Re:SoundStream: An End-to-End Neural Audio Codec by whazor on Sunday August 15, 2021 @02:48AM (#28186725)
I wonder if this approach would discriminate against certain voices.
Re:SoundStream: An End-to-End Neural Audio Codec by jcims on Sunday August 15, 2021 @03:47AM (#28186943)
Almost certainly, but this is true of most low bitrate codecs. I've got a very deep voice and it becomes largely unintelligible in marginal mobile signal conditions. If anything this one might be more tweakable and/or personalizable than what we use today.
Re:SoundStream: An End-to-End Neural Audio Codec by dan-robertson on Sunday August 15, 2021 @06:19AM (#28187637)
To some extent, surely. In their samples they have some music and some audio with background noise. The music survives ok and the clanging of the background noise is reduced to clicking so maybe languages with clicking sounds do ok too.

I suppose there are a few futures:

- the paper was very innovative but nothing really happens in the ‘real world’

- Google roll it out globally to one of their products and we discover that some voices/accents/languages work poorly (or well)

- The same but with slow rollout, feedback, studies, and improvements to try to make sure everyone gets a good result.

Re:SoundStream: An End-to-End Neural Audio Codec by patrakov on Sunday August 15, 2021 @03:17AM (#28186835)
For the previous version of Lyra, there was a concern that it would discriminate against some languages.
Re:SoundStream: An End-to-End Neural Audio Codec by dmitrygr on Sunday August 15, 2021 @12:05AM (#28186180)
For a company the penalizes sites for being "not mobile friendly", they really drop the ball on this blog. All the graphs have their right sides off screen, hidden, with no chance to see them on a normal modern Android phone.
Re:SoundStream: An End-to-End Neural Audio Codec by tyingq on Sunday August 15, 2021 @12:51AM (#28186358)
The pictures and graphs are embedded in an html table for reasons I don't understand.
Re:SoundStream: An End-to-End Neural Audio Codec by ub99 on Sunday August 15, 2021 @12:36AM (#28186303)
Looks cut off in portrait mode but fits fine in landscape (iPhone X).
Re:SoundStream: An End-to-End Neural Audio Codec by sodality2 on Sunday August 15, 2021 @12:34AM (#28186297)
Now this is impressive. Truly quite amazing technology. I pray we reach the day where my 1200GB music collection can be compressed without noticing, to a fraction of the size.
Re:SoundStream: An End-to-End Neural Audio Codec by NathanielK on Sunday August 15, 2021 @02:27AM (#28186645)
Depends what format you have it in now. 160kbps opus is transparent for almost all music. That's 1/5 the space of a FLAC collection, or 1/2 the size of 320kbps mp3, two other popular ways to store music without noticeable loss in quality.
Re:SoundStream: An End-to-End Neural Audio Codec by sodality2 on Sunday August 15, 2021 @10:09AM (#28188841)
I actually do encode to 128 opus and crank up the encoder settings for music synced to my own devices, because I know they can play it. It's pretty much transparent to me but I am not an audiophile with audiophile equipment, which is why I keep the originals.
Re:SoundStream: An End-to-End Neural Audio Codec by PaulDavisThe1st on Sunday August 15, 2021 @12:56AM (#28186372)
Why? that's not much more than 1TB, easily available in various storage formats for a very low price. What's the issue with the size?
Re:SoundStream: An End-to-End Neural Audio Codec by sodality2 on Sunday August 15, 2021 @01:07AM (#28186399)
It's not very portable. I can't keep it on my phone without a 1TB micro SD, and I could theoretically carry an SSD but thats not really portable as it requires internal connection. And if it's an external drive with a USB connection then it's probably much larger than my phone.

It's certainly able to be kept and it's not a particularly unwieldly amount of data for the average power user, but if I ever had to bring my music somewhere (and I mean all of it) I'd be screwed. Plus that data would take hours to copy if ever needed. I spent just an hour today dding a 64GB SD card.

I just keep my favorite albums and discographies in 320 MP3 as I know there is almost no music player on earth that will fail to play it. Then I keep that in a micro SD in my wallet, for that emergency in which you desperately need some Pink Floyd.

Re:SoundStream: An End-to-End Neural Audio Codec by jonplackett on Sunday August 15, 2021 @03:37AM (#28186911)
More likely that storage capacity/$ will scale faster that codec efficiency.

In a few years you’ll have it on your phone anyway!

Re:SoundStream: An End-to-End Neural Audio Codec by f1refly on Sunday August 15, 2021 @04:07AM (#28187007)
1. Wow, what an obsessive music geek you are.

2. Consider https://beets.io for management, it may be just the system you want. One of its features is converting you library into a lossy version while keeping your high res files in a very managed way.

Re:SoundStream: An End-to-End Neural Audio Codec by toast0 on Sunday August 15, 2021 @01:28PM (#28190551)
As you seem knowledgeable, it looks like beets manages music you've already ripped, do you have a suggestion for ripping? Last time I tried, it was a disaster (I got a lot of poor quality audio and ended up nuking everything because it was too much work to listen to everything and try again)
Re:SoundStream: An End-to-End Neural Audio Codec by f1refly on Sunday August 15, 2021 @05:55PM (#28192907)
It's been a while since I ripped my last CD, but I always found https://github.com/bleskodev/rubyripper very useful. It uses cdparanoia to get multiple rips and combines them into what it thinks is the exact information located on the original cd.

These days, I can fortunately download most stuff directly from bandcamp :)

Re:SoundStream: An End-to-End Neural Audio Codec by sodality2 on Sunday August 15, 2021 @10:10AM (#28188852)
Now that's cool. Thank you so much, I'll look into that.
Re:SoundStream: An End-to-End Neural Audio Codec by amonavis on Sunday August 15, 2021 @01:03AM (#28186389)
Personally for me the issue is not the size but the transfer (think streaming, service migration, etc.)
Re:SoundStream: An End-to-End Neural Audio Codec by sodality2 on Sunday August 15, 2021 @01:08AM (#28186405)
I just have a syncthing folder shared between my VPS, my home server, my desktop, laptop, and phone; it syncs it continuously so a download once syncs it everywhere else. It's basically set it and forget it.
Re:SoundStream: An End-to-End Neural Audio Codec by amonavis on Sunday August 15, 2021 @01:44AM (#28186506)
In some countries data consumption is still capped, and reducing bandwidth usage goes a long way in making media consumption more affordable.
Re:SoundStream: An End-to-End Neural Audio Codec by ris on Sunday August 15, 2021 @08:02AM (#28188041)
Notice in the examples how Lyra tries to construct voice-like sounds out of the "music" sample? This is what scares me a bit about these codecs.
Re:SoundStream: An End-to-End Neural Audio Codec by overboard2 on Sunday August 15, 2021 @08:39AM (#28188306)
Why?
Re:SoundStream: An End-to-End Neural Audio Codec by ris on Sunday August 15, 2021 @10:53AM (#28189228)
Because of the possibility of it adding "words" that were never there.

See also: xerox photocopiers changing dimensions of technical drawings with "too clever" compression schemes.

Re:SoundStream: An End-to-End Neural Audio Codec by numpad0 on Sunday August 15, 2021 @12:59AM (#28186382)
How does it compare to Codec2?
Re:SoundStream: An End-to-End Neural Audio Codec by fsiefken on Sunday August 15, 2021 @02:17AM (#28186615)
in low bandwidth situations, for recording speeches or podcasts, i have a similar question. the codec2 examples have a 8 KHz range so can't be compared to the lyra ones as is. perhaps you could encode your own voice wavs with ffmpeg and compare. in this case there's also the question of portability, can the resulting files be played on android or iphone and how much cpu cycles/battery power would it cost? I'd rather listen to 1 hour of lyra speech then codec2 speech if the battery would last twice as long.
Re:SoundStream: An End-to-End Neural Audio Codec by numpad0 on Sunday August 15, 2021 @01:00PM (#28190322)
I don’t know but there seemed to be a demo for Codec2 running on STM32F4, whereas Lyra repository README explains how its optimized implementation allows it to run on a midrange phone in real time, so…
Re:SoundStream: An End-to-End Neural Audio Codec by sanxiyn on Sunday August 15, 2021 @01:47AM (#28186513)
Isn't Codec2 a speech codec? This seems to target general audio.
Re:SoundStream: An End-to-End Neural Audio Codec by remcob on Sunday August 15, 2021 @10:34AM (#28189053)
Lyra is also a speech-only codec, yet is included in the comparison.

Note also that Codec2 had some experimental work extending it with the WaveNet neural network, which improved the performance.

Given both of this, it seems disingenuous to exclude Codec2 from the comparison. I can only assume its left out because it performs well at even lower bitrates.

Re:SoundStream: An End-to-End Neural Audio Codec by ksec on Sunday August 15, 2021 @04:48PM (#28192392)
>Over the past few years, different audio codecs have been successfully developed to meet these requirements, including Opus and Enhanced Voice Services (EVS).

I guess the Google AI team works separately from the main Google Team. Lot of respect for pointing out EVS.

Re:SoundStream: An End-to-End Neural Audio Codec by a-dub on Sunday August 15, 2021 @05:30AM (#28187418)
would be interesting to characterize the behavior of the encoder (ie; how does it differ from a mel warped spectrum... or what is the warping that it learns?)

also would be kinda neat to see something that is pretrained in this way, and then does a small amount of humans in the loop training iterations to see if quality improves or perhaps an uncovering of something previously unknown about human auditory perception...

Re:SoundStream: An End-to-End Neural Audio Codec by miohtama on Sunday August 15, 2021 @07:13AM (#28187831)
> Opus is a versatile speech and audio codec, supporting bitrates from 6 kbps (kilobits per second) to 510 kbps, which has been widely deployed across applications ranging from video conferencing platforms, like Google Meet, to streaming services, like YouTube. EVS is the latest codec developed by the 3GPP standardization body targeting mobile telephony. Like Opus, it is a versatile codec operating at multiple bitrates, 5.9 kbps to 128 kbps.

Why 5.9 instead of 6 kbps? Did I want to have some kind of PR victory over Opus?

Re:SoundStream: An End-to-End Neural Audio Codec by zinekeller on Sunday August 15, 2021 @07:34AM (#28187924)
Re:SoundStream: An End-to-End Neural Audio Codec by PYTHONDJANGO on Sunday August 15, 2021 @05:58AM (#28187554)
Is this good for synced music transmissions or is it another simulation of the 1950s telephone sound?
Re:SoundStream: An End-to-End Neural Audio Codec by aaaaaaaaaaab on Sunday August 15, 2021 @06:51AM (#28187742)
3 kbps is not enough for music in any meaningful way.
Re:SoundStream: An End-to-End Neural Audio Codec by neatze on Sunday August 15, 2021 @05:05AM (#28187294)
Where is a line between end-to-end audio compression and end-to-end audio encryption ?
Re:SoundStream: An End-to-End Neural Audio Codec by dan-robertson on Sunday August 15, 2021 @06:24AM (#28187653)
Well encryption would likely be bad if it made things smaller and compression would be bad if it didn’t make things smaller.