I don’t like it either, but apparently semi-structured data is the future. It’s proving easier to build systems that tolerate bad data and broken references than it is to get our data normalized and cleaned.

I’m the guy on most teams pontificating about SQL and data-oriented code, but even I’ll concede that large scale live systems probably shouldn’t serve from an SQL database.

Do they have only one huge table in the db that contains all of their entities? From the key structure that seems reasonable: shard id, type, object id.

If that is the case, is a relational db still the right place to keep such data? It seems that if the goal is to store this kind of data in one or a few huge sharded tables, a different tool could be optimizing to do the job better.

It seems that I’m missing something…

A pin was a 1.2 KB json blob. There were other tables but pins was the big one. Why MySQL? It did not destroy data like the alternatives.

how storage became efficient https://medium.com/pinterest-engineering/evolving-mysql-comp... https://medium.com/pinterest-engineering/evolving-mysql-comp...

Before I joined in late 2013, they had not known how to run schema change without downtime. Once we fixed the kernel the db’s were nearly completely untaxed in terms of performance. They did however need large instances due to disk usage.

In 2024 all major cloud providers have NoSQL (AWS DynamoDB and its equivalents) that will easily scale to beyond this use case, managed purely by good single table design. That is the best practice for this and most basic use cases — not RDBMS in 2024.

Why? key/value data stores have consistent hashing which scales to planet scale built in. No need for sharding or caching or other bs that brings down slack and GitHub every other month.

Most applications don't need planet scale, though.

do you have a reference to how this might be accomplished? how do you avoid high read / write latencies if you put everything into a single table / key-value pair per user?

I think they're referring to single-table data design as described in https://www.youtube.com/watch?v=HaEPXoXVf2k It's quite clever and scales well, but completely unnecessary for the vast majority of CRUD apps.

IMO it’s also difficult when you’re in the initial phases of a new company/product, because the system is evolving fast and that doesn’t play nice with single-table-design needing upfront design to work well.

I have had much better luck sticking to a plain old relational DB stack for those phases, because it’s fast to iterate, and then consider moving to single-table NOSQL when the system/DDD has “gelled” and traffic is starting to hockey-stick

I know cases where caches on top of a kv store improved performance a lot. I don’t think it’s as simple as you claim.

Like another comment pointed out they seem to store pins as a JSON blob in a single column anyway. They probably should've used Drupal.

Really, this is pinterest. They store 'boards' and 'pins', you may as well have a bit of fun with it. What use cases do they have for the features you list? They probably only used a database for write transactions?

I deeply value database technology and use it daily, but I don't think something simple like this needs to worry too hard about it.