Look at the list of what Apple Sing includes:

Adjustable vocals: Users now have control over a song’s vocal levels. They can sing with the original artist vocals, take the lead, or mix it up on millions of songs in the Apple Music catalog.

Real-time lyrics: Users can sing along to their favorite songs with animated lyrics that dance to the rhythm of the vocals.

Background vocals: Vocal lines sung simultaneously can animate independently from the main vocals to make it easier for users to follow.

Duet view: Multiple vocalists show on opposite sides of the screen to make duets or multi-singer tracks easy to sing along to.

------

The part of the article where they state these things explicitly...

As a matter of fact, calling out "millions of songs in the Apple Music catalog" actually makes it seem like the adjustable vocals will only be available on certain songs that they've added support for.

My guess is that it's entirely dynamic. It's hard to imagine the complexity of doing batch processing to render each song in the library, and maintain that as new songs are uploaded, and update the renders for software improvements. Better to just do it realtime.

And since classical, instrumentals, esoteric ambient stuff, death metal, etc, will probably not be supported by the algorithms, I think the "millions" refers to those that can be processed in realtime.

> [...] an on-device machine learning algorithm that can process music in real time

Their latest apple TV includes the A15 which includes a 'neural engine' for ML, and this is also included in their latest iPhone / iPad, so that might be part of it.

I think this only requires pre-making two audio files per track, and simultaneously streaming these.

Real-time lyrics, Background vocals and Duet view are all nice features too, but the hardest part processing-wise is analysing how loud you sing into the microphone. It's just karaoke with a good UI.

That’s the understatement of the century. “this only requires […] simultaneously streaming these”

They are likely using a sophisticated ML version of what old karaoke machines did, and removing the vocals in real-time.

Source: https://techcrunch.com/2022/12/06/apple-music-is-getting-a-n...

Wonder why they take this approach though, as it is clearly over-engineering (if I correctly understand that the goal is just to make vocals volume adjustable).

Depends what the other non-functional requirements were. i.e. if the NFRs were as follows:

* Cannot increase bandwidth / mobile data usage.

* Cannot impact music quality / bitrate.

* Has to work offline.

* Cannot increase on-device storage.

* Has to be responsive.

Then two audio streams might not work.

Another advantage of doing it on-device is that it doesn't actually change any of the backend architecture too. It might be a lot of change to a lot of systems for a feature which only adds a small amount of functionality - i.e. architecting your entire backend and streaming around seperating audio tracks might not be the right focus.

Since Apple is all about on-device processing with so many of its features, going back-and-forth to the data center doesn't seem to be its style these days.

That's more of a Google thing.

And no one can accuse Apple of telling its advertisers that you start your day with Funky Cold Medina.

Is that the actual reason though? My personal impression has been that it's a combination of reasons that benefit Apple. The increased user privacy being a nice bonus for users, but not the primary reason:

1. Producing phones powerful enough for on-device ML both justifies the high price point to the general public and is a good marketing point (along with increased user privacy)

2. Avoid backend infrastructure costs. Why spend extra money on servers, maintenance, and compliance when we can just offload the work to the devices themselves since they're capable?

3. Bonus: The unplanned obsolescence for new features like the one announced is also a side effect that benefits Apple.

I do not get the impression that Apple's primary focus is to benefit users and their privacy.

It makes it so you have to buy new devices sooner.

The privacy thing is a nice side-benefit and PR thing, but let's be realistic here.

EDIT: Just to remind everyone, we are literally in a thread about a new device feature that is trivial to do in the cloud, which Apple chooses to do on-device, which makes it only a feature for its newest generation of products...

If Apple wanted to support this for user-provided mp3s then on-device would make sense. It doesn't sound like they support that though.