News for hackers. Stuff that matters.
Re:How Figma's databases team lived to tell the scale by ummonk on Thursday March 14, 2024 @06:04PM (#39709487)
This seems to be an architecture Cloudflare is aiming to support with their SQLite service. One database per customer, each database located in the customer’s primary region.
Re:How Figma's databases team lived to tell the scale by jasonwatkinspdx on Thursday March 14, 2024 @06:14PM (#39709568)
I think there's quite a few people chasing similar ideas, like Azure's Durable Entities.

I've been calling it the Lots of Little Databases model vs the Globe Spanning Gorilla.

Like the Spanner paper points out, even if your distributed database semantically appears like a single giant instance, in practice performance means developers avoid using distributed joins, etc, because these can lead to shuffling very large amounts of intermediate results across the network. So the illusion of being on a single giant machine ends up leaking through the reality, and people end up writing workarounds for distributed joins like async materialization.

If we give up the single machine illusion we get a lot of simplification, at the cost of features devs were unlikely to use anyhow. I see having consistent distributed commit but without cross shard joins as a really interesting alternative.

And besides scalability I like the extra security rope of fine grained partitioning from the start.

I'll write a blog post along these lines if I get anything worthwhile done.

Re:How Figma's databases team lived to tell the scale by themoonisachees on Thursday March 14, 2024 @07:23PM (#39710187)
An advantage worth noting is that having actually separated databases means you physically can't make these expensive operations, so a junior dev can't write incredibly inefficient code that would bring down your entire infra.
Re:How Figma's databases team lived to tell the scale by ummonk on Thursday March 14, 2024 @11:35PM (#39711818)
Also makes it a lot harder for devs to do some footgun and leak data across domains.
Re:How Figma's databases team lived to tell the scale by aledalgrande on Thursday March 14, 2024 @11:54PM (#39711912)
Bring down the infra or foot you a 6 figures bill at the end of the month
Re:How Figma's databases team lived to tell the scale by shalabhc on Thursday March 14, 2024 @10:20PM (#39711431)
"Lots of Little Databases" reminded me of https://www.actordb.com/ which does lots of server-side sqlite instances, but the project now looks defunct.
Re:How Figma's databases team lived to tell the scale by ummonk on Thursday March 14, 2024 @11:40PM (#39711848)
Interesting. Durable Entities strikes me as closer to Cloudflare's Durable Objects (both in name and in design as actors backed by persistent storage).
Re:How Figma's databases team lived to tell the scale by fauigerzigerk on Friday March 15, 2024 @04:37AM (#39713369)
That would be fantastic. Unfortunately it's not true. D1 doesn't support the one database per customer approach unless you have just a handful of customers that you can set up manually.

You have to create each database manually using wrangler or the website. Then you have to create a binding for each database in wrangler.toml so that the database becomes accessible as a variable in your Workers code. Then you have to change your Worker source code to do something with that variable. Then you redeploy.

The issue is that Workers cannot create or list databases. There's no API for it.

Re:How Figma's databases team lived to tell the scale by ummonk on Friday March 15, 2024 @01:45PM (#39718537)
They have said they're working on some sort of solution (which they intend to have available by the time D1 exists beta) that will allow some sort of dynamic bindings specifically to address this use case.

In the meantime, though it's not a great solution, one can create and query databases through the REST API instead https://developers.cloudflare.com/api/operations/cloudflare-...

Re:How Figma's databases team lived to tell the scale by fauigerzigerk on Friday March 15, 2024 @03:54PM (#39719983)
>They have said they're working on some sort of solution

That's great to hear, thanks!