News for hackers. Stuff that matters.
Re:How much is Spotify paying Google Cloud? by dmytton on Monday March 07, 2016 @06:36PM (#11242584)
Yes, but that has no impact on the cost. It's a flat rate in 64kb blocks. Batching and compression would only affect the networking fees from Google, not the per API call fees. If you batch up 64 1kb messages into a single batch then Google still bills you for 64 publish API calls.
Re:How much is Spotify paying Google Cloud? by sharpy on Monday March 07, 2016 @06:49PM (#11242635)
They are wrapping small events into fewer larger ones. Hence the word batching.

I have 64 1k messages? Ok, I am going to put them inside a single 64k message and reduce my request count by 64x, and profit!

Re:How much is Spotify paying Google Cloud? by foota on Monday March 07, 2016 @07:19PM (#11242759)
"Batched calls are accounted for as multiple operations." - https://cloud.google.com/pubsub/pricing
Re:How much is Spotify paying Google Cloud? by sharpy on Monday March 07, 2016 @07:58PM (#11242892)
Right.

If you look at the API, the publish API allows you to push multiple messages in a single API call, and if that is the batching - publishing multiple messages - then yes, you and others are absolutely correct that there is no cost benefit.

But rather than publishing

{ "messages" : [ { message 1 }, { message 2 } ] }

You publish

{ "messages" : [ { container message containing both, still less than 64k } ] }

Then you've reduced your operations count by half. Cloud services like this have a per message overhead, for things like auditing and delivery tracking, which is why the pricing model is what it is.

If you are ok with having to unpack and process multiple messages at once as I've described, then it's a way to optimize against the pricing model.

Re:How much is Spotify paying Google Cloud? by dmytton on Tuesday March 08, 2016 @02:04AM (#11243917)
The Spotify graphs from Google's monitoring support the numbers I quoted. They may well be batching but Google is seeing (and billing) the numbers I quoted.
Re:How much is Spotify paying Google Cloud? by foota on Monday March 07, 2016 @09:47PM (#11243264)
That makes sense. Thanks. Do you see anything saying they're doing this?
Re:How much is Spotify paying Google Cloud? by Gigablah on Monday March 07, 2016 @07:58PM (#11242893)
He's talking about a single call containing multiple messages in their own serialization format.
Re:How much is Spotify paying Google Cloud? by i0exception on Monday March 07, 2016 @09:38PM (#11243233)
If 2M requests result in 1 Gb network traffic, each message is 500 bits (~60 bytes). Theoretically, that's 1K messages per batch.

Even if practically that number is much lower (around 100 per batch), the cost ends up becoming around $3,000 for the messages and $9,000 for the network, so roughly $12,000 per month.