News for hackers. Stuff that matters.
Re:How Figma's databases team lived to tell the scale by nosefrog on Thursday March 14, 2024 @02:13PM (#39707181)
Coming from Google, where Spanner is this magical technology that supports infinite horizontal sharding with transactions and has become the standard storage engine for everything at Google (almost every project not using Spanner was moving to Spanner), I'm curious how Figma evaluated Cloud Spanner. Cloud Spanner does have a postgres translation layer, though I don't know how well it works.

It seems like they've (hopefully only temporarily) given up real transactional support with their horizontal postgres scheme?

Re:How Figma's databases team lived to tell the scale by umvi on Thursday March 14, 2024 @02:28PM (#39707350)
Never a good idea to rely on Google proprietary tech (unless you are Google)... it could be sunset at any time without warning. I use GCP but I try my best to stay Google agnostic (avoid GCP-only offerings, etc) so that I can move to AWS if Google pulls the rug out from under me.
Re:How Figma's databases team lived to tell the scale by weitendorf on Thursday March 14, 2024 @04:13PM (#39708445)
I'm biased having worked on GCP, but I think GCP actually has a very good track record of not sunsetting entire products or removing core functionality. When I worked on AppEngine, I would often see apps written 10+ years ago still chugging along.

It is true though that GCP sometimes sunsets specific product functionality, requiring changes on the customers' part. Some of these are unavoidable (eg committing to apply security patches to Python 2.7 given that the rest of the world is mostly not upstreaming these patches anymore), but not all of them.

I would certainly use Cloud Spanner externally now that I've left the company, and IMO spanner has such a compelling featureset that it's a strong reason to use GCP for greenfield development. The problem though is that it's only available on GCP, and could become expensive at large scale.

Re:How Figma's databases team lived to tell the scale by jerrygenser on Thursday March 14, 2024 @05:33PM (#39709235)
>It is true though that GCP sometimes sunsets specific product functionality, requiring changes on the customers' part. Some of these are unavoidable (eg committing to apply security patches to Python 2.7 given that the rest of the world is mostly not upstreaming these patches anymore), but not all of them.

A good example is probably IoT. I've heard first hand anecdotes of very difficult migrations off this service.

Re:How Figma's databases team lived to tell the scale by nojvek on Thursday March 14, 2024 @08:52PM (#39710908)
My previous employer was a GCP customer. Google did pull occasional shenanigans totally breaking us with random upgrades without notifying us. Their support wouldn’t acknowledge it was their fault until we were mad at them.

My newer employer is AWS. Their offerings are a lot more stable and support is helpful.

If you want to build a serious business I would avoid GCP. Google doesn’t really give a shit at winning Cloud. Ads is their core business.

Re:How Figma's databases team lived to tell the scale by weitendorf on Thursday March 14, 2024 @10:11PM (#39711389)
Ugh I'm so sorry, you should definitely have been notified in advance about breaking changes, unless they were accidental bugs and regressions. That was something we took pretty seriously for the products I worked on.

I have definitely enjoyed using AWS support. But I've also encountered some broken/janky functionality (eg using custom domain + api gateway + websockets + Lambda for the apparently niche-task of having a real website use the Lambda-websockets integration) on AWS that I don't think would have made it to production at Google. I also really dislike how they handle project-level logging compared to how it's done on GCP.

Ultimately I do think some GCP products like Bigquery, Cloud Run (I'm biased here), and Spanner as well as general DevEx/reliability factors are compelling enough for GCP to be worth serious consideration, even if AWS offers better support.

Re:How Figma's databases team lived to tell the scale by rockostrich on Thursday March 14, 2024 @03:26PM (#39707960)
GCP products have a much better track record than Google consumer products when it comes to support since there are usually enterprise customers with multi-year contracts worth tens, if not hundreds, of millions of dollars using them.
Re:How Figma's databases team lived to tell the scale by callalex on Thursday March 14, 2024 @03:50PM (#39708223)
That’s what AppEngine customers thought.
Re:How Figma's databases team lived to tell the scale by weitendorf on Thursday March 14, 2024 @03:57PM (#39708293)
I worked on AppEngine (well, Serverless Compute) at Google for over 4 years and left on Friday. Did something happen to AppEngine in the last week?
Re:How Figma's databases team lived to tell the scale by callalex on Thursday March 14, 2024 @04:10PM (#39708416)
I’m talking about the pricing disaster that happened in 2017/2018 where user prices went up from 10x-100x because Google wanted to kill the product without actually killing it.
Re:How Figma's databases team lived to tell the scale by tempnow987 on Thursday March 14, 2024 @07:11PM (#39710094)
Didn't they also hammer folks using google maps in a business / cloud context?

I was also an old app engine user, but bailed ages ago (original app engine).

Re:How Figma's databases team lived to tell the scale by jerrygenser on Thursday March 14, 2024 @05:32PM (#39709226)
IoT is one example of a big backbone service that was sunset.
Re:How Figma's databases team lived to tell the scale by endisneigh on Thursday March 14, 2024 @05:47PM (#39709348)
It had barely any usage though from what I can tell from searching about it.

Not that it’s any solace to those affected.

Re:How Figma's databases team lived to tell the scale by marcinzm on Thursday March 14, 2024 @06:22PM (#39709639)
AI Platform is a recent example that’s been deprecated.
Re:How Figma's databases team lived to tell the scale by rockostrich on Thursday March 21, 2024 @02:49PM (#39782767)
Wasn't it just superseded by Vertex AI? According to the banner announcement in the docs: > All the functionality of legacy AI Platform and new features are available on the Vertex AI platform.
Re:How Figma's databases team lived to tell the scale by opportune on Thursday March 14, 2024 @03:37PM (#39708083)
My perspective from working both inside and outside of Google:

The external spanner documentation doesn’t seem as good as the internal documentation, in my opinion. Because it’s not generally well known outside of G, they ought to do a better job explaining it and its benefits. It truly is magical technology but you have to be a database nerd to see why.

It’s also pretty expensive and because you generally need to rewrite your applications to work with it, there is a degree of lockin. So taking on Spanner is a risky proposition - if your prices get hiked or it starts costing more than you want, you’ll have to spend even more time and money migrating off it. Spanner’s advantages over other DBs (trying to “solve” the CAP theorem) then become a curse, because it’s hard to find any other DB that gives you horizontal scaling, ACID, and high availability out of the box, and you might have to solve those problems yourself/redesign the rest of your system.

Personally I would consider using Cloud Spanner, but I wouldn’t bet my business on it.

Re:How Figma's databases team lived to tell the scale by nojvek on Thursday March 14, 2024 @08:54PM (#39710939)
If you really have that much data and traffic, the $ costs start to add up to multiple engineer comp costs. At that point it’s cheaper to move to something you have good control over.

I.e sharding at application layer and connecting to the DB instance replica where the customer data is hosted.

Re:How Figma's databases team lived to tell the scale by weitendorf on Thursday March 14, 2024 @10:41PM (#39711539)
Depends. The cost may pay for itself but the engineers you have already may have higher ROI things to do. It's also nice to have operational stuff managed for you. Personally I'd be happy to pay extra for the kinds of problems Spanner solves to free myself up to do other things (to a point, ofc).

> sharding at application layer and connecting to the DB instance replica where the customer data is hosted.

Spanner does global consistency/replication. If having good performance per-tenant globally is a concern, this helps a lot, and is hard to implement on your own. It can also ultimately save you money by limiting cross-region traffic.

Re:How Figma's databases team lived to tell the scale by shalabhc on Thursday March 14, 2024 @02:37PM (#39707441)
Global consistency is expensive, both latency-wise and cost-wise. In reality most apps don't need global serializability across all objects. For instance, you probably don't need serializability across different tenants, organizations, workspaces, etc. Spanner provides serializability across all objects IIUC - so you pay for it whether you need it or not.

The other side of something like Spanner is the quorum-based latency is often optimized by adding another cache on top, which instantly defeats the original consistency guarantees. The consistency of (spanner+my_cache) is not the same as the consistency of spanner. So if we're back to app level consistency guarantees anyway, turns out the "managed" solution is only partial.

Ideally the managed db systems would have flexible consistency, allowing me to configure not just which object sets need consistency but also letting me configure caches with lag tolerance. This would let me choose trade-offs without having to implement consistent caching and other optimization tricks on top of globally consistent/serializable databases.

Re:How Figma's databases team lived to tell the scale by foota on Thursday March 14, 2024 @04:27PM (#39708566)
While it doesn't help much with the costs of replication, Spanner can be configured with read only replicas that don't participate in voting for commits, so they don't impact the quorum latency.

Reads can then be done with different consistency requirements, e.g., bounded staleness (which guarantees data less stale than the time bound requested).

See https://cloud.google.com/spanner/docs/reference/rest/v1/Tran... or https://cloud.google.com/spanner/docs/reads#read_types and https://cloud.google.com/spanner/docs/create-manage-configur...

Re:How Figma's databases team lived to tell the scale by eatonphil on Thursday March 14, 2024 @02:56PM (#39707635)
See also: "Strict-serializability, but at what cost, for what purpose?"

https://muratbuffalo.blogspot.com/2022/08/strict-serializabi....

Re:How Figma's databases team lived to tell the scale by ksb on Thursday March 14, 2024 @02:22PM (#39707289)
Ping time from AWS data centers to GCP ones
Re:How Figma's databases team lived to tell the scale by hipadev23 on Thursday March 14, 2024 @02:18PM (#39707235)
The problem is nobody outside Google trusts them to run or operate anything.

Edit: To the Googlers downvoting these comments. Your behavior only reinforces our views.

Re:How Figma's databases team lived to tell the scale by szundi on Thursday March 14, 2024 @02:21PM (#39707281)
Google means: good chance discontinued after you have worked out the bugs and have a stable system at last
Re:How Figma's databases team lived to tell the scale by groestl on Thursday March 14, 2024 @03:07PM (#39707758)
This is definitely not the case with their core cloud products.

> almost every project not using Spanner was moving to Spanner

This even includes Datastore. Even Datastore moved to Spanner.

Re:How Figma's databases team lived to tell the scale by echelon on Thursday March 14, 2024 @02:15PM (#39707202)
Why would anyone marry themselves to Google? That sounds like the most boneheaded move possible.

First, you should never be beholden to a single vendor for the most critical technological underpinnings. You're backing yourself into a corner.

But more importantly, Google can't even figure out how to prioritize their cloud efforts. That's not a good partnership to be in for anyone except Google.

I wouldn't care if my solution was 10x worse than Cloud Spanner from a technology perspective. It'd be 1000x better from a strategy perspective.

You can hire engineers to do consistency at scale. It's your core competency and you can't just handwave and outsource that, lest you wind up stuck in a crevasse. Hire smart engineers and do the work yourself. It'll pay off.

Re:How Figma's databases team lived to tell the scale by Thaxll on Thursday March 14, 2024 @05:13PM (#39709057)
You can't, that's why spanner only exists a Google. That tech is that good, not found anywhere else.
Re:How Figma's databases team lived to tell the scale by echelon on Thursday March 14, 2024 @06:33PM (#39709739)
But you don't need it. There are a limitless number of ways to deal with the problems spanner solves. And by choosing your own, you can focus on the subset that matter to your case and not be tied down to Google.
Re:How Figma's databases team lived to tell the scale by zenbowman on Thursday March 14, 2024 @05:23PM (#39709152)
I wouldn't say "infinite", its still susceptible to read hotspotting; and while fine-grained locking enables generally higher write throughputs, you can still get in a situation where interconnected updates end up being pretty slow.

That said, its way better than anything else I've used in my career.