News for hackers. Stuff that matters.
Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @11:36AM (#24746487)
Seems like an obvious and sane target for antitrust legislation - compare with e.g. the mobile number portability directive in the EU way back in 2002.

Related opinion piece:

> Data portability: An antitrust weapon for the digital economy? (Le Monde, 2018)

http://www.fondapol.org/en/the-foundation/medias-en-en/data-...

> In concrete terms, what would portability in the digital economy look like? Let us imagine we could easily transfer our playlists between online music platforms (Spotify, Deezer, Apple Music, etc.), our files between cloud service providers (Google Drive, DropBox, OneDrive, etc.), our purchase records between online retailers (Amazon, CDiscount, Zalando, etc.) or our social graph between social media platforms (Facebook, Twitter, Snapchat, etc). By limiting platform lock-in, such measures would significantly intensify competition within these market segments.

Re:A Note about Spotify Transfers by eadz on Sunday October 11, 2020 @07:41PM (#24750078)
The irony is that Spotify is a partner in the app store fairness coalition (https://appfairness.org/).

I consider their developer platform[1] to be an app store of sorts, and they do not play fair when it comes to holding onto their market power in certain areas.

Anyone have a contact in the App Fairness Coalition to raise these points with?

[1] https://developer.spotify.com/community/showcase/

Re:A Note about Spotify Transfers by ec109685 on Sunday October 11, 2020 @10:18PM (#24750850)
Yeah, seriously. There isn’t some higher ideal — every company is in it for themselves and then works on justifying the policy that will further their interests through such coalitions.

No doubt that the same company that sends a cease and desist to a developer who provides an export feature would act the same way as Apple if they were in its shoes.

Re:A Note about Spotify Transfers by wayneftw on Sunday October 11, 2020 @12:26PM (#24746912)
According to Spotify’s own site [0], their users have

> ...the right to request a copy of your personal data in electronic format and the right to transmit that personal data for use in another party’s service...

[0] https://www.spotify.com/us/legal/privacy-policy/#s3

Why can’t a transfer service just use that file?

Perhaps, it doesn’t include playlists or liked artists/albums/songs?

Re:A Note about Spotify Transfers by tzs on Sunday October 11, 2020 @01:07PM (#24747274)
I've just requested my data. When I get it I'll take a look and see if it contains playlist information and either edit this comment or post a reply to myself if the edit window has closed with the results. They say it can take up to 30 days to process.
Re:A Note about Spotify Transfers by morsch on Sunday October 11, 2020 @01:32PM (#24747502)
Here's a previous discussion of someone getting the full dataset: https://news.ycombinator.com/item?id=17681289

Tried the GDPR data export from Spotify. By default, you get like 6 JSON files with almost nothing. After many emails and complaining and a month of waiting, I got a 250MB archive with basically EVERY INTERACTION I ever did with any Spotify client, all my searches. Everything.

Re:A Note about Spotify Transfers by tzs on Wednesday October 14, 2020 @04:36PM (#24781294)
I got the results. It contains playlists.
Re:A Note about Spotify Transfers by DaiPlusPlus on Sunday October 11, 2020 @03:19PM (#24748277)
> They say it can take up to 30 days to process.

What the heck...

My current day-job is a multi-tenant SaaS system - every single entity in any of our heterogenous databases is tagged with a tenant identifier (it's also used in composite PKs/FKs in RDBMS databases to prevent incorrect inter-tenant references). Every database system we use (MSSQL, MySQL, Cassandra, etc) has some way to query itself (INFORMATION_SCHEMA, system_schema, etc) which allows for query-generation to generate queries that will reliably dump all data we have associated with any tenant (and by extension, any user) within seconds, even if there's been some ad-hoc or unplanned database design changes (this is the only time I use "SELECT *" in production code!) - this is from a weekend project where I wrote a simple web-application that generates and runs those queries and dumps data to CSV files for tables and JSON for document-stores, shoves them into a zip and uploads them to Azure Blob storage. I didn't build it for GDPR compliance but actually to allow me to undo any unintentional data-deletion without needing to do a full database restore.

My operation's scale is nothing compared to Spotify - but if a tool I built in a couple of days can reliably dump all data about a subject across multiple independent cloud databases within seconds - what's Spotify's excuse?

Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @03:47PM (#24748506)
It's obviously not a technical problem. The 30 days comes from the GDPR legislation. Spotify is just behaving like an obnoxious child that needs a good smackdown from the EU. Howevever the GDPR also stipulates that this needs to happen without "undue delay".

I'm not a lawyer specializing in EU law, but I think this wording was put there for the precise reason of being able to punish companies that behave like that.

They'll get to them eventually.

Re:A Note about Spotify Transfers by jrockway on Sunday October 11, 2020 @06:22PM (#24749609)
I don't think that's fair. If the law sets the limit at 30 days, there is no reason to do it sooner than 30 days. It is obviously a money-losing proposition to do it quickly, and the only reason it's an option at all is because of the law. I guess that makes it a reasonable compromise -- everyone is equally unhappy, but the world is a slightly better place.

The reality is that "30 days" translates to "someone will run a script manually once a month" whereas something like "100 milliseconds" would force you to have automation. It's a reasonable tradeoff for rare events. There is a lot less engineering effort required, and that makes it cheaper.

Re:A Note about Spotify Transfers by viraptor on Sunday October 11, 2020 @10:19PM (#24750852)
> there is no reason to do it sooner than 30 days

There is. Any tiny issue puts you past 30 days and invites a slap from the regulator. There's a lot of space between 100ms and 30 days. If you can't run your monthly process twice a day instead for example, you're most likely delaying on purpose - that's the "undue delay" part.

There's a company I know which decided not to pay any invoices until the last possible moment - which one day resulted in the electricity being cut to an office building with hundreds of people. Do you think that was a good decision in the end?

Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @06:28PM (#24749656)
You are reacting on just one part of the directive, even though I just told you about you another part. Seriously? What's the point?
Re:A Note about Spotify Transfers by SpicyLemonZest on Sunday October 11, 2020 @08:34PM (#24750386)
This seems like an extraordinarily hostile inference to draw from Spotify's statement about the maximum amount of time it'll take. Do you have some specific reason to think Spotify is engaging in undue delay?
Re:A Note about Spotify Transfers by toast0 on Sunday October 11, 2020 @03:45PM (#24748486)
As someone else mentioned, there's a legal maximum, and I think a company is also held to their quoted timeframe (and they may be required to quote a timeframe as well).

Additionally, having a delay can be useful to make it harder for someone to take over your account, and grab all your data; especially if there's sensitive data that might not be otherwise accessible or enumeratable through the UI or API.

Re:A Note about Spotify Transfers by draaglom on Sunday October 11, 2020 @03:30PM (#24748365)
The 30 day figure is likely derived from the legal limit under GDPR for a Subject Access Request (one calendar month).

Why commit to a tighter SLA than you legally have to?

Re:A Note about Spotify Transfers by ViViDboarder on Sunday October 11, 2020 @05:11PM (#24749134)
Exactly. I’d be surprised if it actually takes 30 days, but there’s no point to them quoting something faster.
Re:A Note about Spotify Transfers by tracer4201 on Sunday October 11, 2020 @06:39PM (#24749732)
This is a very naive view "If I can do <X> in Y time, why can't <insert some company>?".

Any large company has a tens, hundreds, or even thousands of different teams who may own a system with some data that identifies you as a customer. Presumably, they're on the hook for serving you (the customer) all of this data when you ask for it. Honestly, I'd hate to be a legal counsel at a company having to sift through every attribute/data column trying to figure out what we "have" to return vs. what we can probably keep hidden as a trade secret, but I digress.

Anyway, there's no guarantee that even half of the systems storing this data were designed with GDPR (or whatever privacy-related) compliance in mind.

Consider a system that's storing nested JSON blobs with customer-identifying data several layers deep. You happen to be on a team that owns this mission critical system. Your legal department gets you to prioritize some dev work to build a system to quickly extract this data.

You'll (probably) do it in the most cost effective manner – it might mean rearchitecting your system if the cost of extracting that data is very high. It could be that you have the tools to extract this data very quickly and so you just need to plug and play. Or there's at least one more scenario where such an operation is so expensive (and impacting on your main business function) that you accumulate a bunch of requests (i.e. 30 day window) and run some ETL to get all the customer data and respond on the compliance requests. So with that approach it's definitely not a real-time or near real-time response.

And my example scenarios here are actually pretty simplistic for a large company. Imagine the scenarios where you have N customer records with some loose notion of an evolving schema over the years. You're not even sure how to query that data or transform it... or your automation works 98% of the time to pull the customer data but 2% of the time it fails, and so you have to time (legally) to manually have an engineer fix that edge case (depending on how expensive that work is) or the engineer manually goes and pulls your data, accounting for whatever edge case that was discovered.

The more data you store about a customer, the "harder" this would get.

I would probably use data retention policies to drop as much data as I can, although I'm sure at a large company, your business customers push back: "Oh we've never had to use that data set, but we want you to keep it because we might use it to build features for some new ML model in the future or to solve other problem <Y>".

Re:A Note about Spotify Transfers by DaiPlusPlus on Monday October 12, 2020 @01:08PM (#24756650)
You’re not wrong (and I’ve upvoted you too) - but given the importance of complying with the spirit of the GDPR and other consumer-first regs - especially given that Spotify is HQ in the EU they should be doing more. You do set-out a reasonable explanation for non-immediate data-dumps, but from the evidence I believe companies like Spotify and many others intentionally drag their feet and do the bare minimum, at least in-part, to discourage requests. Of course that also makes it bloody obvious too. More telling is the fact that they aren’t being transparent about it either. If they posted a simple statement explaining in-not-too-much-technical-detail why it takes so long that would be a start.
Re:A Note about Spotify Transfers by input_sh on Sunday October 11, 2020 @01:09PM (#24747306)
I've done that. Ignoring that it takes a few days to receive a dumb, what you get is just a subset of data.

I've been a Spotify user for over half a decade, but the oldest date I could find in the dump was a little over two years ago.

I then contacted their support asking for all data, but I've made a mistake of contacting them with a wrong email address (which lead to them being unable to find my account) and I haven't bothered since.

In the best case scenario, I'd say it takes a week to get the entire dump, which is not very convenient for switching to a competitor.

Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @12:29PM (#24746934)
I haven't tried doing so with Spotify, but basically it's typically too complicated and slow (see other comments) to be practically useful for the typical consumer of today.
Re:A Note about Spotify Transfers by lxgr on Thursday October 15, 2020 @03:47AM (#24785960)
It doesn't include any standardized/machine-readable identifiers.

For example, the Spotify API track-level responses always contain the ISRC [1], but the JSON files contained in the GDPR request don't.

Spotify clearly does only the bare minimum required by the law and even actively frustrates GDPR requests by taking the maximum allowed time to respond to them.

[1] https://en.wikipedia.org/wiki/International_Standard_Recordi...

Re:A Note about Spotify Transfers by actuator on Sunday October 11, 2020 @11:58AM (#24746659)
Also, it runs afoul of sections of GDPR.

I don't agree that Spotify has to let users transfer Spotify curated playlists as that is a business strength. Songshift seems to show Spotify curated playlists on their homepage like Discover Weekly so I can see why Spotify won't like this.

But if my understanding of GDPR is right they have to allow users to transfer playlists created by themselves.

Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @12:07PM (#24746737)
Right. Forgot that.

Regarding GDPR and data portability:

In practise you can expect to receive a (link to a) zip file via email with a bunch of XML documents after going through a convoluted dance. Also, it often takes some time (days) to receive the data. Basically, nothing that can easily be automated by a third part service - you need user auth of the requesting user to access the data.)

"You must comply with a request for data portability without undue delay and at the latest within one month of receipt of the request"

I guess it's because data portability isn't trivial, and it was wasn't the key aspect of GDPR.

As a contrast: The EU mobile number portability legislation says: max 1 working day.

Re:A Note about Spotify Transfers by iav on Sunday October 11, 2020 @09:09PM (#24750537)
Carriers pay to port numbers to a neutral third party that maintains the Number Portability service. In the US, that used to be $6 per number port (which is basically just a DB row update operation) but a new vendor has reduced it by more than half. Carriers view that expense as pocket change as subscriber acquisition cost is well over $100 in total.

Without a legislative mandate and a third party integration partner that assures the port is successful, seems unlikely that a market leader like Spotify would ever allow them to exist.

Re:A Note about Spotify Transfers by actuator on Sunday October 11, 2020 @12:17PM (#24746815)
Yeah, data portability is hard to define as a strict standard as it is extremely product specific.

Even when we implemented it for my last company, we did JSON exports on signed S3 paths. I thought structured data was the easiest way for any competitor or user to use their data.

Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @12:23PM (#24746866)
I feel they could have included a clause that would let a designated third party access the extracted data on the requester's behalf in an automated way.

I think it could have worked pretty well to keep it as vague as that. (The auth aspects would need more detail, obviously.)

I don't think data format fragmentation is a big issue (just implement adapters for the top five competitors in your market...), but rather the impossibility of doing it in a quick and seamless manner on behalf of someone wanting to migrate their data to your service from some other service.

Re:A Note about Spotify Transfers by actuator on Sunday October 11, 2020 @02:53PM (#24748087)
> I feel they could have included a clause that would let a designated third party access the extracted data on the requester's behalf in an automated way.

Yeah, this would have been nice. I am not sure if legally you could enforce that as automation itself is vague and depends on the source and target both.

> I don't think data format fragmentation is a big issue (just implement adapters for the top five competitors in your market...), but rather the impossibility of doing it in a quick and seamless manner on behalf of someone wanting to migrate their data to your service from some other service.

I think if companies were being honest in their efforts for data portability, as a first step they would not try to reap the benefit of the 30 day window by as much as they can.

For most, it should take mins or hours to get the export yet the best turnaround I have seen from any service has been from Tinder, it was I think 3 days.

Re:A Note about Spotify Transfers by watermelon0 on Sunday October 11, 2020 @06:03PM (#24749481)
This really depends on the service. In my experience, Netflix and Facebook took only a few hours to export the data.
Re:A Note about Spotify Transfers by sokoloff on Sunday October 11, 2020 @08:57PM (#24750478)
> just implement adapters for the top five competitors in your market

Is there anything to prevent the export file format from shifting regularly? You know, for ongoing product improvement reasons, naturally.

Re:A Note about Spotify Transfers by saagarjha on Sunday October 11, 2020 @04:48PM (#24748968)
Which ones? GDPR guarantees "right to data portability", not "right to an open API". I'm sure they could give you a CSV if you personally write to them and still be compliant.
[dead]
Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @02:33PM (#24747948)
Lol. A jingoist.
Re:A Note about Spotify Transfers by jtbayly on Sunday October 11, 2020 @03:06PM (#24748200)
I’m pretty sure it’s a violation of community standards to use a person’s country of origin as an insult or slur. Are you intentionally trying to denigrate a people group?
Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @03:08PM (#24748208)
I apologize. I edited the comment accordingly.
Re:A Note about Spotify Transfers by jtbayly on Sunday October 11, 2020 @03:12PM (#24748223)
Lol. Nice.

Anyway, let me know when the GDPR actually helps with something instead of just making the internet more of a usability train wreck.

Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @03:18PM (#24748268)
I'm pretty sure it has helped a lot already.

The key part is the "the don't keep data around forever just because you can" part.

Sure, the GDPR got a bit bloated, but I honestly believe most europeans lives got a little bit better because of it.

Edit: I think you're thinking about that horrible cookie warning law. That one needs to burned to the ground.

Re:A Note about Spotify Transfers by yoavm on Sunday October 11, 2020 @03:58PM (#24748590)
It's not even just European lives that got better because of GDPR, we're all benefiting from having big companies forced to comply with it. I have used the "Right to erasure" multiple times with American companies just by saying I'm from Europe.
Re:A Note about Spotify Transfers by tpmx on Sunday October 11, 2020 @04:21PM (#24748776)
Awesome hack :) How are they going to begin to validate the you're a EU citizen?
Re:A Note about Spotify Transfers by fuzxi on Sunday October 11, 2020 @05:15PM (#24749162)
It's probably easier for most companies to just comply with all requests than to bother with verifying the citizenship of each requester.
Re:A Note about Spotify Transfers by milofeynman on Sunday October 11, 2020 @03:00PM (#24748147)
This one is clearly anti-competive and I hope someone will help them out and pick it up. This is the most obvious requirement for the future of preventing these monopolies. The ability to get your data out.