News for hackers. Stuff that matters.
Re:ULIDs and Primary Keys by kevincox on Thursday December 29, 2022 @06:21PM (#34179468)
On the other hand if you ever want to shard based on ID full randomness is fantastic. There is a funny tradeoff where if all of your IDs are sequential then you get very good locality in your storage system. But that backfires if it creates too much of a hotspot.

Maybe the best option is something like a UUIDv7 (time + random) ID and you can tune your shard keys by shuffling the bits. Maybe you want something like moving the last 4 bits to the front of the shard key so that you have 16 hot spots, which is good for performance without any being too hot.

Re:ULIDs and Primary Keys by tremon on Thursday December 29, 2022 @06:35PM (#34179599)
What's the use case of sharding on a technical key? I thought most sharding is done on business keys (e.g. per customer, region or year)?
Re:ULIDs and Primary Keys by crazygringo on Thursday December 29, 2022 @07:04PM (#34179851)
In my experience sharing is generally done on a "technical" key because it's easy to arbitrarily partition into any desired number of shards.

Something like per-year could result in different years having vastly different disk usage requirements, which would be a headache.

While per-region or per-customer isn't really sharding at all in my understanding. It's just giving each customer (or region) their own dedicated/private database.

Sharding is generally about dividing a single table (for business logic purposes) among (usually) equally-sized servers for performance/capacity reasons, according to some function of the primary key (modulus, hash, etc.).

Re:ULIDs and Primary Keys by kevincox on Thursday December 29, 2022 @07:14PM (#34179924)
A customer ID could well be a UUID. And sharding by year is similar to sharding by a UUID or ULID with a timestamp prefix.

But yes, in what dimension you shard will usually depend on the application.