News for hackers. Stuff that matters.
Re:Sharding Pinterest: How we scaled our MySQL fleet by halayli on Wednesday August 19, 2015 @02:17PM (#10087386)
This looks like a big hack to compensate for using the wrong tool. Cassandra would have been a better solution IMO.

With Cassandra, you can set replication factors, speed up the writes, and automatically shard the data without having to manage your own "mapping tables".

Re:Sharding Pinterest: How we scaled our MySQL fleet by econner on Wednesday August 19, 2015 @02:19PM (#10087399)
Cassandra was immature when this work was started (late 2011). The team was also much more familiar with the ins and outs of operating mysql.
Re:Sharding Pinterest: How we scaled our MySQL fleet by richardwhiuk on Wednesday August 19, 2015 @02:31PM (#10087479)
There's also no transactions in Cassandra, although that's questionable useful given that the unidirectional map may be on a different machine.
Re:Sharding Pinterest: How we scaled our MySQL fleet by brianwawok on Wednesday August 19, 2015 @07:22PM (#10089189)
Cassandra has very lightweight transactions.

What he lists as his transaction use case (update if unmodified) I do right now with Cassandra and an IF clause and a timestamp...

i.e. update foo set x = y AND last_modified_timestamp = 456789 IF last_modified_timestamp = 12345

see http://www.datastax.com/dev/blog/lightweight-transactions-in...

Re:Sharding Pinterest: How we scaled our MySQL fleet by halayli on Wednesday August 19, 2015 @02:30PM (#10087467)
Technical debt is not an excuse to use the wrong tool. This is how you end up having a hammer.

Cassandra was mature by late 2011 and we were using it in production back then (billions of records).

Re:Sharding Pinterest: How we scaled our MySQL fleet by luckycharms810 on Wednesday August 19, 2015 @02:45PM (#10087581)
Technical debt doesn't necessarily arise from using the wrong (a.k.a pretty similar but with a few features less than what you need) tool. In my experience, it's more likely that technical debt arises from trying to use thew newest tool (which looks to be a silver bullet) rather than trusting your own experiences. It's surprising to me that people are willing to move to a completely different technology rather than extend and repurpose what your engineers are already familiar with.
Re:Sharding Pinterest: How we scaled our MySQL fleet by joantune on Thursday August 20, 2015 @11:03AM (#10092289)
Although, by using an existing tool, as long as it fits your needs, you'll avoid reinventing the proverbial wheel, ofcourse at the expense of learning something new.. which might not be a bad thing..
Re:Sharding Pinterest: How we scaled our MySQL fleet by econner on Wednesday August 19, 2015 @02:39PM (#10087536)
The mapping tables are used to store relationships between objects, not to define shard locations. The shard mapping is encoded in the object id.

In any case, I would have a knee jerk reaction to not trust any database system that hit 1.0 a month before I wanted to start using it, especially for mission critical core data.

Re:Sharding Pinterest: How we scaled our MySQL fleet by acveilleux on Wednesday August 19, 2015 @02:45PM (#10087583)
Doubly so when everything is catching fire around you from how overloaded it all is.
Re:Sharding Pinterest: How we scaled our MySQL fleet by wereHamster on Wednesday August 19, 2015 @04:54PM (#10088427)
For this very reason many projects nowadays jump directly from 0.x to 2.x :) 2 is the new 1.
Re:Sharding Pinterest: How we scaled our MySQL fleet by _up on Wednesday August 19, 2015 @02:47PM (#10087598)
But if you also need to use MySql anyway, Cassandra would be the "wrong Tool". Because you would have to maintain two DBMS instead of one. You want to minimize complexity and this can mean implementing your own Sharding.
Re:Sharding Pinterest: How we scaled our MySQL fleet by nemothekid on Wednesday August 19, 2015 @03:26PM (#10087815)
Even today I'm not sure I'd recommend Cassandra for his use case. A stated requirement he needed was:

>Support asking for N number of Pins in a board in a deterministic order (such as reverse creation time or user specified ordering). Same for Pinner to likes, Pinner to Pins, etc.

This can be a pain to model in Cassandra. It would require denormalizing for every key you wanted to order on, and it generally makes updating data a pain.

Re:Sharding Pinterest: How we scaled our MySQL fleet by jlongtine on Wednesday August 19, 2015 @04:01PM (#10088058)
I don't think that the schema here allows you to do this kind of query even in MySQL. They are throwing everything in that blob, which makes it pretty difficult to sort on anything that's in the blob. So, I think they would have to do what you are describing, denormalize everything... or get all the data and do the sort in the app. Which is precisely what would be required with Cassandra or another NoSQL system. Am I missing anything that would invalidate what I've just said?
Re:Sharding Pinterest: How we scaled our MySQL fleet by brianwawok on Wednesday August 19, 2015 @07:24PM (#10089195)
If you only need 1 ordering, not too hard in cassandra. If you need a few different orderings, you would need a side table - which seems to be what you need here with mysql.
Re:Sharding Pinterest: How we scaled our MySQL fleet by meritt on Wednesday August 19, 2015 @03:25PM (#10087804)
While that would have made scaling easier they would then run into a different set of massive problems because Cassandra isn't a relational database.
Re:Sharding Pinterest: How we scaled our MySQL fleet by brianwawok on Wednesday August 19, 2015 @07:25PM (#10089200)
But neither is a sharded database. You can't do fancy joins and groups across 400 shards. (Well you can, but you are writing your own code to split up requests and stick it back together.. which is the same thing you would need to do in Cassandra).
Re:Sharding Pinterest: How we scaled our MySQL fleet by georgiecasey on Thursday August 20, 2015 @02:01AM (#10090210)
> 'We had several NoSQL technologies, all of which eventually broke catastrophically'
Re:Sharding Pinterest: How we scaled our MySQL fleet by axiak on Wednesday August 19, 2015 @04:33PM (#10088247)
They also use HBase for a lot of their non relational data.