One of these I struggled with right away was “how do I ensure the front end and back end have consistent data sourcing and validation”, e.g. for various forms, dropdown selectors, etc. I have found surprisingly few resources for general patterns here. So I did your anti-pattern of having an endpoint for each little thing (e.g. /api/v1/country-codes or something like that).
This has felt… ok at best. One challenge, perhaps entirely imagined, is coherent namespacing. Part of the backend is also a public (internally) API. We auto-generate all our docs with swagger, and I was finding that the few meaningful endpoints I had were diluted by the mess of individual endpoints for the frontend. So I threw these under /api/v1/config. Feels like a junk drawer but at least it cleaned things up.
I like the idea you’re proposing here for having one API per page, I think it makes a lot of sense. It also eliminates the “I guess I’ll add another endpoint” and changed it to “let me add another field”. Feels more granular and scoped. I will be giving this a try!
I also have all context on vision for direction of the web app, and field all user feedback and requests. Front end team has no context on user needs, and does not keep any roadmap for the app. So it seems like the above recommended pattern could make sense in that I could drive more front end work from the backseat, so to speak.
However, for long-term scalability, you might run into issues with this approach. If so, take a look at "Bounded Contexts," a technique in Domain-Driven Design that helps break down your application into manageable, scalable units.
1) basic CRUD API for your objects (eg DRF generated model-API in Django). Really easy to build. Client is assumed to have some magic knowledge (eg which country code style you use). Ideally as much of the smarts as possible is server-side, but early on you can compromise on this.
2a) for reads, notice you need nesting / joins, add nested fields. Maybe notice these joins are slow and make them optional; build some expanded=true or with_subobject=true type query params.
2b) for writes, notice that CRUD is getting hard to maintain as frontend can create n^2 possible states for n fields, and backend validation needs to consider all other fields for each update. Move complex state transitions to mutation/action/verb endpoints instead of PUT(any fields). Add your first “status” fields.
This gets you quite far, likely to be fine for 2 years and beyond. After this there are lots of options depending on needs, and overhead associated with more layers/abstraction;
3) use something structured like GraphQL, or just refactor the API to do away with directly updating objects. Maybe lean into Finite State Machines if you have a small number of objects with complex transitions, endpoints trigger those parameterized transitions.
Another option to consider is a server-side rendered framework with client-side hooks like Django+HTMX, Rails Hotwire, Elixir Livewire, where the frontend devs are working on styling, design, templates, and most/all the actual app logic is backend code. This gets you surprisingly far these days. If you are the one defining all the features, you’ll iterate quicker in this mode.
For 2b, could you elaborate a bit? Is this like having dedicated handling for specific functionality? If I understand you, for example instead of PUT-ing some status on /myobject/{id}/, I could have something like /myobject/{id}/update_status/. And the benefit here is assuming that setting status is not a straightforward transition (maybe requires other transformations, etc), that this lets you better compartmentalize this complex logic instead of having one giga-endpoint for all PUTs?
For writes, a typical rest approach as described in the grandparent is good.
Using specific actions such as `update_status` vs. allowing any fields to be supplied is… I think both can work. But it's not easy to cleanly organize the latter. It might be important to split different groups of related fields into separate logical paths, as though they were indeed supplied via separate endpoints. In Ruby/Rails I had success doing this via modules included into the model. The advantage of letting any fields be updated in any combination is that this would allow you to build different interfaces (i.e. admin UI, user UI, using REPL console if your language supports it) with different combinations of fields, and expect them all to work. You don't have to do that if you don't anticipate this need yet, but it's also true that splitting this logic into different urls is not all that different from splitting its handling under the hood, while allowing all fields submitted at once.
On 2b, yep exactly as you say. Instead of updating arbitrary fields with PUT/PATCH, you would have an endpoint per action. (Or a mix of both approaches if it is more expedient).
This doesn’t make much sense for “data bag” objects with lots of potential fields to edit like a user’s profile. But say your object has a mutation that always takes one parameter to update a model with many other incidental fields, you can map that to a POST /obj/<id>/update_foo {param} which is basically an RPC mapping to a single function.
This makes most sense when you have objects that look like FSMs since in that case you might have a few transitions, each taking different args, and each dependent on a specific state as precondition. But I have used this for e.g.“users/<id>/activate” which updates the status, sets a updated_at timestamp, and cannot be undone. The alternative is something like PATCH the is_active field, and having some gnarly logic to handle bookkeeping around time stamps and preventing reverting.
Another way of thinking about it is if any combination of values is valid, it doesn’t help. But as soon as you need to handle state transitions and particularly when you have multiple, then simply updating raw fields results in extremely messy validation logic within your update endpoint.
Stated another way, when building a tool, don't let information about the incidental context leak into the design of the tool itself. It should be designed to serve its intended purpose independent of when, how, where, or why it was built. This way, it won't break when that context changes, or when it's used in a new context.
This is impossible and in many cases undesirable. As long as the purpose and behavior of the function is clear, it is OK if it has contextual roots. Actually, no, it isn’t just OK, it is the way it must be.
Take one aspect of a sorting function for example. Should it be stable? Or not? This decision is rooted in the context. To try to make the function independent of context is impossible and unwise.
Are you making a different point?
Your comments about stable sorting are well and good, but they don’t disprove what I was trying to get across with my example. Why a ‘business logic’ function uses a stable sort (or does not) is contextual.
I understand your point, but do you understand mine? This is just one example. The idea for mutually beneficial communication is this: one person strives to be charitable and understand another’s example as a way of understanding their point. This is not a skill necessarily taught or nurtured in the field of software engineering, but it should be.
Most of us have the deductive/analytical skills to use an example to prove our point. It is actually harder to take someone else’s example and see their point. It is harder empathetically and computationally; it requires some inductive steps. (I think there’s a connection between empathy and generalization/induction, but that’s for another time.)
My comments are obvious — almost tautological — if you think epistemologically. Programs and functions are designed to solve problems in context. You can’t get around this fundamental truth. Philosophy has been around longer than programming; in fact, it is one of the historical roots of mathematics and programming. Discussions of good design, which may involve parsimony, orthogonality, and more, have been discussed at length well before software existed. We’d be wise to not ignore lessons already learned.
Such is my experience, having seen and learned a lot from a lot more than software. Claiming that something is context-free is a bold claim. Pure mathematics and logic are probably the only areas than can make that argument credibly.
So every feature/change only needs one engineer.
Primarily, this philosophy encourages you to maintain a strict connection between needed use cases, and the code that enables them. Once that connection is severed, your code expands beyond the required use cases, and you end having to maintain support for phantom/theoretical use cases.
Instead of just implementing ACLs/permissions on existing APIs you have to develop and maintain two APIs, one for the customers (that will get outdated with lot of missing features and be less battle-tested) and one “real API”.
Our public API is fit for the use-cases of our big customers that want deep integrations, and our frontend’s API is able to handle our specific workflows.
We have a bunch of shared backend code, but the endpoints that our API customers use are very specific. Sometimes we build custom endpoints for a single customer which would be useless for our frontend/services. Keeping them separate allows them to gain needed functionality without trying to bolt it on to existing internal APIs, and cluttering it all into one messy blob.
Of course our internal API is a mess in a bunch of other ways but at least we don’t have to deal with the public parts making it even messier
Mobile app: this is addressed in the article.
Naming: Usually most naming for different parts of pages is handled by the design(er), but even if not, once you have an approximate naming convention for parts of the site, naming becomes easy. And the stakes for these names aren't that high, because you are only scoping everything down to a single page, so global consistency and precision in naming is not that imporant. It just needs to be enough for someone to understand what it most likely refers to on this page.
Every API call was so form fit to the UI that nearly every UI tweak needed frontend and backend changes.
It didn’t need to be this way (there was a lack of engineering skill and leadership), but even past the surface level much of the backend code beyond the API interface also became form fit to the specific use case meaning many times seemingly simple changes required near total reimplementation of all of the business logic and other underlying code.
End of the day, showing the same data or exposing the same flows elsewhere in the application required nearly a total reimplementation on the backend. This created an explosion in size of the codebase and complexity.
This whole idea seems to basically be going back to RPC-based APIs and all the drawbacks that led to everyone kind of collectively agreeing REST was better (even if most people only sort of half implement it).
Our approach, staring down a roadmap of projects that basically involved exposing the same functionality over and over with different pictures in front of it was to steer _hard_ the other direction.
We sat down and put together an API specification that laid out the general shape of an API. Not the specific resources, but for any given collection/resource how it should be exposed. We found common “tough” use cases and figured out up front how to expose those RESTfully and documented the patterns to maintain consistency. We put down a framework in code to make compliance with the spec the “easy” path for most cases.
Then we approached the rest of the design from an API perspective, not a frontend or backend perspective. For any given flow/operation/etc, what would a good interface look like for this. (This is very similar to figuring out what the interface for a module/class/etc should look like, given its responsibility, before writing it.) Then we built it.
The end result has been that more and more frequently requests and even entirely new projects require _no_ backend involvement. Often the FE team doesn’t even need to ask any questions—given any endpoint that exposes a collection of resources, they know _exactly_ how to work with it because they’re all the same.
When new business rules or requirements come in, we no longer need to apply them in 5 or more different places across two codebases. And we’ve still maintained a good separation of frontend and backend duties to the point our FE team is busy but kind of bored—they’re only really concerned with the view/display logic at this point.
Company history is largely lost to the mists of time, but the API routes were all prefixed `/v5/`. I can only imagine how much further along the company would be if they’d spent the last decade building on top of a foundation instead of rebuilding the foundation every two years as they have been with the RPC model.
Having to change backend code is not necessarily a bad thing. Just don't structure your code as two seperate codebases built by two different teams.
I'm trying to understand how this is possible. Couldn't the backend logic that contributes this response fragment in one endpoint be extracted and reused to contribute the same response fragment in another endpoint?
Was backend not modularized and reused?
The way I understand your approach, you ended up building and maintaining CRUD resources similarly to how they would exist in a naive public API, but the difference is that you spent extra time to build more meaningful, rich, business-level resources than serving nearly raw database records.
This is probably a good compromise that isn't disagreeing with the article as much as you seem to be implying. The difference between your approach and the article's approach is a small side-step. You either let the front-end fetch your rich business-level resources separately, or you package them into pages and serve them in a nice single package. Or you can build pages as per the article, and extract these rich resources that you display in those pages into separate endpoints for some additional flexibility. Either way, you end up with rich use-case driven resources, packaged differently.
You are basically saying we should take a name that conveys info about its content (companyName) and change it to something that contains none (pageTitle).
I can tell if a field is being used for a page title by looking at the html, I can’t tell what it contains unless it is named well. Additionally if I ever want to hop into the backend my time on the frontend will have given me 0 context if you are renaming all the fields.
Also if you want to use Typescript this would prevent a bunch of automated type generation you could do.
The result with this strategy is that you may need the companyName and end-up at some point with code that does back:
companyName = pageTitle
or worse:
companyName = pageTitle.split(‘ - ‘)[0]
Because there is a moment when you will need to get the companyName again for some comparison or something whatever logic (e.g. is selected account profile current company ?)
This feels very confusing.
Instead pass companyName and keep it that way until the very last moment that you display it on the page (e.g. when rendering the view/template)
If the page only makes sense if a company name is used as a title, then of course use companyName as the variable.
The idea is to write things as standalone components. Which is a pretty universal programming strategy.
On the one hand, you have some data coming in format X.
On the other hand, you need the data in format Y.
What you need is to define how to map X to Y. And then when you request Y, a way to indicate what parts of X you really need, so the code getting X can optimize, if it needs.
In my experience, the real issue is how to cache the X - should we be caching all of it depending on the Y, or should we be caching parts and understanding how they fit together. I have had problems with updating caches, and I am looking for solutions!
The problem with caching is that it is very often highly individual and it usually comes with drawbacks, i.e. you win performance here but lose it there or you win performance but you lose consistency. (related: CAP theorem)
So I doubt there is any "generic" solution that will or even could make you happy. But if you find one, let me know. :-)
I think reactive programing solves the "dependency graph" problem to you in the sense that you just have to "change" one value and then all dependencies (and their dependencies) update. You still have to define the relations though.
What I see here is that the "view" is split across the front-end and the back-end. Or in other words, the back-end contains both the model and the JSON API as a view. I think this is what is meant by "I suggest you stop treating your frontend as some generic API client, and start treating it as a half of your app."
I still want a clean separation between the model, e.g. companyName, and the view, e.g. pageTitle. Somewhere, there needs to be the explicit logic (in the back-end part of the view) that says that the the pageTitle for page A is the companyName. It's okay for many different views to "reuse" fields from the model, but each view should have its own front-end and back-end components. That the view is split across a front-end/back-end divide through the nature of the web-stack doesn't change the fact that each view should remain distinct from the other views.
So I can agree in the sense that the alternative where you create a general-purpose JSON API as a model and view in the back-end, and then essentially code up an additional model and view for each front-end page/element with logic to convert the API data into the necessary content is a waste of time.
If back-end provides the entirety of data to build the page, then you limit front-end decisions to UX based on the specific data, and no longer allow them to pull in new functionality or foundational business logic over to their side.
Perhaps to argue this requires munging "view" with "front end"?
MVC needn't mean database schema, front end, and logic. It could, but then what is a "view" in the database?
Perhaps you're reinventing MVVM?
In some CRUD situations though, they can be nearly the same and you can convert the domain model into a specific view model with some kind of adapter (how people do this varies a lot). This adapter will take the end user security level into account, for example. Also, to avoid sending lots of useless data for a list, you can have a list view model. But if you do that, be sure to either keep the two in sync, or you have one and keep track of whether it’s partially or fully hydrated.
The MVC or mvvc or whatever paradigm is somewhat orthogonal to the technical back end/front end.
For example, if you remove a field that's still needed, you'll receive a type error, eliminating the problem of losing track of use cases.
If you think about functions or object constructors in general, it kind of makes sense. One of the most important architectural practices I've ever discovered is: don't pass in the parameters that you have, pass in the parameters that the function needs. So if you have companyName, and you are constructing a page that has a title, you shouldn't pass it companyName: companyName, you should pass it pageTitle: companyName (parameter name: parameter value). It's crucial that the parameter is named after what is needed, not what you have. This is kind of the essence of what the article is suggesting.
If your front-end is reusing a lot of fields from back-end, it's likely that you're passing it what you have in your database/resources, and not what it needs — fields for constructing the UI, configuring the views. Moreover, I rarely see front-end going through the exercise of defining exactly what they need. The article encourages more of that.
Once the front-end starts reusing the generic back-end fields everywhere, you lose track of your use-cases. Everything you ship from the back-end becomes potentially important for mysterious reasons that not even front-end can easily understand anymore. Front-end already built complex dependency trees based on these generic fields. If you actually untangle these trees, it might turn out that your front-end can only exist in 5 different "modes", but you can no longer see that in the tangled mess. You can no longer streamline and optimize these 5 configurations.
So to summarize: pass what is needed, not what you have is a good general architectural rule that's also applicable between back-end and front-end.