Just like people put some PHP scripts on fully hosted Apache+MySQL 20 years ago. This was very common and far easier than AWS. (And reliable, too, although not as scalable, but the needs then were different). The point being that all of this has been here before. Every few years some complexity creeps back in (in exchange for some other benefits) and then it‘s eliminated again and some progress is made. But works always expands.
Recently I helped a friend who is a teacher with her Excel sheets to do grade reports. Pretty well done for Excel, yet it was a terrible user experience. Even if there are only 1000 users, working with this 1 hour per month, a proper custom made software would have been better and easily economically viable. Even no-code has existed for a long time, but it never fits perfectly.
Similarly, people regularly complain about just glueing components together. As opposed to what? Copying sorting algorithms right out of a CS class? It‘s a strange idea. Looking at the code I work with, over the years, I find very little glue. True, there are abstractions and sometimes they get in the way. But they are there for a reason. Whether you start out from scratch or use frameworks, much of the application will revolve around business-related data structures.
You can give it a try: Take your real-world product, strip out the abstractions, replace the built-in UI widgets, sorting routines, hash tables of your language and maybe that OR-Mapper and your GraphQL-server framework and so on with your custom code and something minimalistic. It won‘t take that much time and code and compared to the stuff on top you‘ll find it‘s not that much that you actually used in the end. Nothing to glue together anymore.
Not that it makes sense to do this. But the idea that glueing things together has replaced „real“ development is very much mistaken.
As opposed to, I'm guessing, adapting Monte-Carlo tree search to Go and inventing AlphaGo. Or what BellKor was doing back in the Netflix Challenge days. Not copying sorting algorithms out of a CS class, but solving a puzzle with a clever new algorithm that just works, and then everything else falls into place.
(Call it "if you write it, they will come" taken to its logical conclusion.)
The point was that actual software engineering has not been more or less trivial than a few decades ago and that the glue isn‘t that much after all.
There was never a time when the daily work of software engineers was, in fact, research. There are plenty of research institutes and universities and even research labs of software companies where you can do this after you‘ve got your degree. Just go there, many of my friends are doing exactly that.
It's not going to eliminate work in software development, it's going to increase work in developing software systems by increasing the average value of each unit of work, and, simultaneously, move the average level of the work higher up the abstraction ladder, just as every advance in software productivity since “what if we didn't have to code directly in machine code and instead has software that would assemble machine code from something one-step more abstract”.
I’d love to exchange notes with you on the lessons you learned building your system and the challenges you faced.
I’m using rancher/k8s with docker on top of “unreliable” hosts with AWS/GCP/DO/Azure providing “spill over” capacity for when those unreliable cheap hosts prove why they’re unreliable.
Is it possible we could get in touch? You can reach me at hnusername at Google’s mail service. Would love to connect if you’re open!
Take a look at AirTable and IFTTT/Zapier.
Its not going to save us from all of the work, but it does eliminate a lot of redundant work. For sure.
> Even the example of hosting complexity being replaced by cloud companies seems kind of silly to me. Maybe that’s saving very small companies a sizable fraction of their engineering resources, but I really doubt it for medium or larger companies.
I think that hundreds of billions of dollars has been spent moving from localized IT to cloud. Do you really believe that was all a waste of money? For example, most of those medium or large companies had their own operations software backends, and most of it was eaten by clouds services/APIs.
> You still need people who understand things like docker, kubernetes, endless different database options, sharding, indexing, failover, backup, message queues, etc. Even if the pieces are now more integrated and easier to put together, the task of figuring out how the pieces will interact and what pieces you even need is still outrageously complicated.
Docker/K8s is a good example. I spent more than a year building a Docker orchestration/hosting startup (eventually decided not to try to compete as an individual with Amazon). But when I recently needed a reliable way to host a new application and database, I did not have to configure Docker or K8s at all. Why? Because I used AWS Lambda and RDS. Those are examples of software eating software. AWS can handle all of the containers for you if you do it that way.
As far as failover and backup, that was handled by checkboxes in RDS. I did not need a message queue because that was built into the Lambda event injection service.