News for hackers. Stuff that matters.
Re:How Spotify ran a large Google Dataflow job for Wrapped 2019 by kossTKR on Tuesday February 18, 2020 @03:51PM (#22360278)
Yeah it's pretty interesting that they undertake this huge task when one of the basic features still doesn't work.

Simply put when you shuffle from all of you liked songs you will mostly get the same tracks over and over - some tracks will stay hidden forever, - pretty weird and annoying.

It seems to stem from issues in relation to this post, ie. sql queries and caching to prevent too much CPU use on their end.

Re:How Spotify ran a large Google Dataflow job for Wrapped 2019 by 2bitencryption on Tuesday February 18, 2020 @04:20PM (#22360519)
I think the root cause is because spotify shuffle isn't true "shuffle" in the mathematical, random sense.

They perform some analysis to increase the "perceived randomness" - e.g., if the true random seed picks the same artist twice in a row (totally possible), pick another song by a different artist, or else people will perceive the shuffle as not "random" enough.

Unfortunately I don't have the source for this right now, but I'm sure someone will hop in and provide it if I'm wrong about this :)

Re:How Spotify ran a large Google Dataflow job for Wrapped 2019 by claudiulodro on Tuesday February 18, 2020 @05:45PM (#22361228)
They have also further modified the shuffle algorithm within the last year or two to favor putting songs at the top that the user hasn't listened to a lot. There are definitely a variety of heuristics involved with their shuffling algorithm.
Re:How Spotify ran a large Google Dataflow job for Wrapped 2019 by downerending on Tuesday February 18, 2020 @05:16PM (#22360968)
I'm familiar with the idea. Their custom algorithm seems to do the opposite. The order actually being generated has very little perceived randomness, far less than what a true random shuffle would look like.
Re:How Spotify ran a large Google Dataflow job for Wrapped 2019 by sorenjan on Tuesday February 18, 2020 @04:55PM (#22360800)
Re:How Spotify ran a large Google Dataflow job for Wrapped 2019 by downerending on Tuesday February 18, 2020 @05:17PM (#22360975)
Amusingly, the comments at the bottom are from a large number of others also noting that their algorithm doesn't work as described.
Re:How Spotify ran a large Google Dataflow job for Wrapped 2019 by on_and_off on Tuesday February 18, 2020 @08:59PM (#22362447)
I worked at another music streaming company, we had to do the same.

That was hugely frustrating, but we would get user reports of the random button being buggy when e.g. the user gets 2 tracks of the same album/artist one after the other.

Of course that can happen if we truly randomize your content !

So we switched to a pseudo random algorithm that tries to have consecutive tracks from different album/artists.