News for hackers. Stuff that matters.

A comment on Streaming data in Postgres to 1M clients with GraphQL.

Re:Streaming data in Postgres to 1M clients with GraphQL by maxpert on Sunday October 16, 2022 @11:56PM (#33229680)
While I love ingenuity of developers at Hasura (because I've been personally through these scaling challenges), I always get a gag reaction with GraphQL. I've honestly tried hard to digest it, and I can tell you at large scale where single DB won't cut it, you would either need to develop a large federation layer like [Netflix](https://netflixtechblog.com/how-netflix-scales-its-api-with-...), or just rip-it out. Streaming might elevate the problem a little, but I real problem still lurks under the hood. The fact that front-end community wants to fit everything under GraphQL bothers me, because every backend developer knows that a single tool/technology is usually not the best tool for solving all problems of your life. Remember the golden words, THERE IS NO SILVER BULLET!
[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by nrb on Monday October 17, 2022 @12:30AM (#33229820)
If database capacity is your main concern and not independent schema deployability, then federation is overkill. You can just connect to whichever databases contain your data in your resolvers within a single service.

You have to be at pretty massive scale before federation becomes necessary and by then (if ever) your frontend teams have experienced benefits that are pretty much miraculous. The reason frontend wants to fit as much as possible into it is because it's vastly better than what came before it unless you have a 95th percentile org that is really doing an outstanding job managing the API via other means.

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by arcticfox on Monday October 17, 2022 @12:05AM (#33229714)
I also have issues with GraphQL at times, but mostly because I find the syntax unintuitive.

What about the concept doesn't work? It's just a syntax for queries, I'm confused why it wouldn't scale.

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by eurasiantiger on Monday October 17, 2022 @07:25AM (#33232096)
It’s not just a syntax for queries—GraphQL is essentially a single-stop shop for entity resolution.
[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by lakomen on Monday October 17, 2022 @08:55AM (#33232802)
I've been thinking about the problem and the more I think about it, I tried many different alternatives, it's best to have plain SQL being sent for queries.

The problem there is whi has the right to run which queries?

So the real problem is authorization.

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by dsandip on Monday October 17, 2022 @12:45AM (#33229891)
FD: I work at hasura and work with users/customers.

Fair point about the DB scaling but not sure if everyone is going to run into this issue. Also, lots of solutions are emerging for this specific problem (with different trade-offs of course) like distributed databases (crunchy, YugaByte, Spanner, etc.). Most folks I work with get by with a reasonably sized DB and some read replicas.

Not a GraphQL problem though IMO.

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by scaryclam on Monday October 17, 2022 @07:37AM (#33232196)
In my experience, if you have more than a small complexity to your data, and you're a medium scale business (or are going in that direction), you're going to end up with this issue. And then it becomes really painful to get out of the situation as migrating data and splitting things up is a real PITA.

While you are correct, not everyone is going to end up with this issue, those that are thinking of getting to a medium sized business should be working to avoid it, which unfortunately means solutions such as hasura lose value. It would be good to see more ability to collect data from multiple sources (please reply and correct me if you already do this, I'm not super familiar with the service).

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by tango12 on Monday October 17, 2022 @08:12AM (#33232470)
Hasura supports multiple data-sources and creating relationships across multiple data sources.

Sources can be databases (postgres family, sql-server, big-query currently, and as we add support for new databases) or REST APIs or GraphQL APIs.

This composition allows in a final GraphQL API that federates across different sources.

From our docs: https://hasura.io/docs/latest/data-federation/data-federatio...

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by mirzap on Monday October 17, 2022 @12:58AM (#33229956)
With Hasura you can create a federation of databases and expose them as graphql or rest endpoints. You can also wrap your existing REST/Graphql services. Almost like API gateway, place where you unify your services, manage access / row-level permissions etc.
[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by e12e on Monday October 17, 2022 @03:52AM (#33230827)
I think graphql with a single db is an odd thing to gain popularity - as the raison d'être for graphql was to federate and proxy different, heterogeneous data sources (databases, json/rest services, soap, other graphql services...).

Speaking of Netflix - I think they had an alternative Api federation service that used some clever tricks with json string vs number keys to allow for alternating http put/get - and through that leverage http level caching. But I can't find the the link...

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by IanCal on Monday October 17, 2022 @06:10AM (#33231588)
The main thing I see it being used for is field selection and model nesting. From an end user perspective, it's pretty nice for this I think. Certainly nice that if I know it's a graphql endpoint I've already got a fairly solid idea about how to query it.
[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by ngrilly on Monday October 17, 2022 @08:58AM (#33232817)
GraphQL raison d’être was not to federate and proxy different, heterogeneous data sources. It has initially been developed as a better API for the Facebook monolith. I asked one of GraphQL’s coauthors directly: https://twitter.com/ngrilly/status/1317415232717860866?s=46&....
[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by e12e on Monday October 17, 2022 @09:08AM (#33232933)
Interesting - I didn't even know that there was a monolith at Facebook [ed: beyond there being reports that they use a large monorepo for code]. I stand corrected, thank you.

This doesn't directly address if graphql at Facebook federated different data sources (eg graph and sql databases) - but I suppose not?

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by ngrilly on Monday October 17, 2022 @09:30AM (#33233191)
I don't have any "insider" information regarding the data sources used by the Facebook monolith, written in PHP, when GraphQL has been developed, but I would expect it to talk to multiple/different datastores, and not just MySQL. Then yes, it could be argued that it has been used to federate multiple data stores, even though it was already the case before using GraphQL :)
[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by myvoiceismypass on Monday October 17, 2022 @12:34PM (#33235978)
Are you talking about Falcor? https://github.com/Netflix/falcor
[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by e12e on Monday October 17, 2022 @02:47PM (#33237925)
I believe so; I think there was a blog post that went into the details - but I can't find it - but I did fint this comment:

https://news.ycombinator.com/item?id=10076235

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by throwingrocks on Monday October 17, 2022 @07:46AM (#33232264)
> The fact that front-end community wants to fit everything under GraphQL

Maybe you’re too out of touch. This isn’t true, at least not anymore. GraphQL is losing appeal.

[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by lakomen on Monday October 17, 2022 @08:52AM (#33232768)
It was never the solution to begin with, IMHO. It's lacking a permissions or let's say authz framework. It's only half the solution. And it's too complicated and there's no real standard, as in real world standard. Everyone has their own little soup cooking, because the manifest is incomplete and always evolving.
[ Parent | on HN ]
Re:Streaming data in Postgres to 1M clients with GraphQL by papito on Monday October 17, 2022 @09:41AM (#33233334)
The front-end (read: Node) community is fairly young. They are still to learn that complexity kills. The hard way.

As for GraphQL - it's great for clients. As a backend engineer, you still have to do the work and a LOT of it.

This is just like microservices. No due diligence on whether the added complexity and destroyed productivity is worth it. "Everyone else is doing it".

[ Parent | on HN ]