Re:How much is Spotify paying Google Cloud?
by jamesblonde on Tuesday March 08, 2016 @12:44AM
(#11243737)
This article misses the biggest cost Spotify have at GCE - analytics. Spotify has migrated a 90 PB Hadoop cluster running on 2000 servers to GCE. That cluster was running Hadoop jobs (mostly Hive) 24x7. The YARN job queue was always full. They now have to pay for the storage and processing costs for that. How much will they pay? I don't know, but I know from their history they are very good at getting discounts on infrastructural software (e.g., their Hadoop software was licensed from Hortonworks and they got an incredible price, because Hortonworks needed a reference customer - sound familiar?)
[ Parent
| on HN ]
Re:How much is Spotify paying Google Cloud?
by dmytton on Tuesday March 08, 2016 @01:54AM
(#11243889)
I could only write about pub/sub in this article because that's the only service we have a full enough set of numbers for. I hope Spotify will be releasing more stats in their future blog posts about the other GCP services they'll be using.
Re:How much is Spotify paying Google Cloud?
by jamesblonde on Tuesday March 08, 2016 @05:35AM
(#11244517)
I have the stats from late January (at Stockholm Hadoop User Group). It's 90PB of data, running jobs with 95 TB of RAM, 40K jobs/day (approximately 12,000 CPUs with probably 60-90% utilization).
Re:How much is Spotify paying Google Cloud?
by jamesblonde on Tuesday March 08, 2016 @05:36AM
(#11244527)
I should add they also have ~100 servers streaming the music to clients for songs that aren't handled by the CDN.
Re:How much is Spotify paying Google Cloud?
by gedrap on Tuesday March 08, 2016 @02:38AM
(#11243980)
I am wondering, how does one migrate huge amounts of data like this?
Re:How much is Spotify paying Google Cloud?
by orthecreedence on Tuesday March 08, 2016 @04:00AM
(#11244203)
Sometimes a truck full of hard drives is the fastest/cheapest way. Not saying that's what they did, but it has been done before.
Re:How much is Spotify paying Google Cloud?
by jamesblonde on Tuesday March 08, 2016 @05:31AM
(#11244497)
Yes, that's how the England 100,000 genomes project is transferring 10s of petabytes from San Diego to England. I assume it was the same here.
Re:How much is Spotify paying Google Cloud?
by JupiterMoon on Tuesday March 08, 2016 @09:09AM
(#11245273)
If any of the data to be migrated is music then much of that is probably somewhere inside google's datacenters already..