For my own work, I took things in a different direction: I made a query language that parses as legal GraphQL using a vanilla parser, but has directives like `@filter, @recurse, @optional` etc. Instead of returning a giant fully-nested result, it flattens results and emits them row-by-row like a SQL database. This means the query evaluation can be lazy and incremental — if you write a query that has a billion results but only load 20 of them, then only 20 rows' worth of work happens.
My company has been using this in production for 6 years now across everything from TB-scale SQL clusters with X00,000 tables/views, to querying our own codebase, configuration, and deployment information to find and prevent bugs. I gave a 10min talk at a conference about this recently, if you'd like to learn more: https://www.hytradboi.com/2022/how-to-query-almost-everythin...
Project GitHub: https://github.com/obi1kenobi/trustfall
The pagination point is described well in nearby comments. It only applies when attempting to paginate across more than 1 dimension at once, like "get all pages of comments in an issue, and all pages of reactions for each comment".
Is paginating across multiple dimensions possible via the REST API?
It seems to me like the main friction that you and others are getting at is that GraphQL is more work to use than REST because you have to write a query. That's a fair point! Perhaps we could publish "canned" queries that are the equivalent of the most commonly used REST endpoints, or make them available for use in the API with a special param.
I worked a bit with GraphQL in the past, but never all that much. Now, I'm sure I could figure it out of I sit down, read the basics of GraphQL, familiarize myself with GitHub's GraphQL schema, etc. But ... it's all a lot of effort and complex vs. the REST API; even with a solid foundation in GraphQL there's still lots more parts.
GraphQL is kind of like giving people a schema to an SQL database and telling them that's an "API"; it kind of is, but also isn't really. There's a reason almost all applications have some sort of database layer (API!) to interact with the database, rather than just writing queries on the fly whenever needed.
There's a divide between at least two types of persona here. On one side is integrators building products and features on top of the GitHub API. For these people GraphQL is arguably superior since the learning curve is manageable and in exchange for scaling it you can make an extremely specific query that returns your product's exact data requirements. The cost of writing this query is amortized over the lifetime of your integration.
On the other side are e.g. users automating some part of their GitHub workflow to save themselves time. I can see how the REST API feels like a better choice here, it's certainly simpler to get started with.
For what it's worth, here[0] is an example of using the `gh` CLI's graphql feature to print a list of all repository URLs for a given organization by login, sorted in a relatively complicated way. It's more verbose than doing this with the REST API but significantly more flexible. This could just as easily be done with curl but as others have pointed out, pagination requires a minimal level of logic to implement, so it's more convenient to use an existing helper. This output gets flushed 10 lines at a time as pages come in, making it suitable to compose with other commands using pipes.
$ gh api graphql --paginate -f query='
query($endCursor: String) {
organization(login: "rails") {
repositories(first: 10, after: $endCursor,
orderBy: { field: STARGAZERS, direction: DESC }
) {
nodes {
url
stargazers {
totalCount
}
}
pageInfo { endCursor hasNextPage } # needed for auto-pagination
}
}
}' -q '.data.organization.repositories.nodes.[]'
# output follows
{"stargazers":{"totalCount":50643},"url":"https://github.com/rails/rails"}
{"stargazers":{"totalCount":5298},"url":"https://github.com/rails/webpacker"}
{"stargazers":{"totalCount":4849},"url":"https://github.com/rails/thor"}
{"stargazers":{"totalCount":4075},"url":"https://github.com/rails/jbuilder"}
# snip
{"stargazers":{"totalCount":3},"url":"https://github.com/rails/hide_action"}
{"stargazers":{"totalCount":2},"url":"https://github.com/rails/gem-buildkite-config"}
{"stargazers":{"totalCount":2},"url":"https://github.com/rails/sqlite2_adapter"}
{"stargazers":{"totalCount":1},"url":"https://github.com/rails/fcgi_handler"}
curl https://api.github.com/graphql -H "Authorization: bearer $TOKEN" --data @query.graphqlDead comment.
You don't need to write a different graphql query, use variables. Good graphql APIs will expose a start and limit field for Pagination.
You don't want to repeat these fetches in the followup queries, they're redundant, and assuming the API is rate-limited they will decrease your query budget for no value.
That counts double if you're fetching multiple paginated fields (which also adds to the awkwardness).
Your pagination is now two-dimensional, with each node's start/limit as points on its own axis.
Now add a third node to the query. Now you have three-dimensional pagination. This quickly goes off the rails.
Try writing a generic N-dimensional paginator for such a query to see why it's difficult. Even designing a sensible and reasonably flexible API for one is a headache.
TBF you could also deal with it using fragments I think, but still, not great.
Most of the simple ways of doing it with SQL are problematic. In these docs
https://docs.spring.io/spring-batch/docs/current/reference/h...
there is some discussion of the problems and some answers that go back to the mainframe era.
But very often, a result missing or duplicated in a few queries isn't a showstopper. On that case, pagination is very simple.
That is you are "full scanning" and making a report or doing something like a reconciliation process in a financial institution, it is not like some image boards where there is a link to the 1781th page of images but it spends forever loading it if you click on it.
I usually write my queries in GraphiQL (check some boxes) and then paste them into the app after I have them working right.
Or maybe TLDR "dear next generation of engineers: SQL is actually pretty good".
That was definitely true before Postgres got row security (admittedly in 2016, after GraphQL was released in 2015), but these days there's really no need to run an entire app server in front of your database just to implement permissions.
I guess it's better when the tooling you use has direct gql integration and builds the queries for you?
Because in my experience accessing the github APIs with "basic" HTTP libraries is way more annoying using v4 (graphql) than v3 ("rest") — it could also be that github's v4 API is dreadful mind, I wouldn't be surprised.
GQL should be more efficient because it's not returning 95% of garbage I don't need, but having to write 5-deep queries (because of the edge indirections) by hand is way more of a pain in the ass than performing two GET requests with a few parameters munged in the URLs. And then I still have to go and retrieve the information 5-deep in the resulting document.
Pagination is also awkward, because now you probably want multiple different queries (and thus multiple different resulting documents) so that your 2+ fetches don't retrieve unpaginated information you got the first time around. And it gets worse when nesting comes in.
I don't think graphql is generally a great experience when you consume it either.