I don't think this is the controversial part or the difficult conclusion, the challenge is how exactly to do that.
Like I said, I spend a great deal of time, on each of my modules. Often, the testing code far outweighs the product code. Documentation is also a big deal.
I write code and documentation for an audience of one (Yours Truly), but I publish it for everyone. It’s not my concern, whether or not the projects get any GitHub stars. In fact, I prefer it, if they don’t.
Here’s an example. I’m currently writing an SDK to provide cleaned data for ML, so I can play around with ML, but it will be quite some time, before I get to that point. I’m still working on making the SDK rock-solid. I have spent the last couple of days, just working on the parser. I still need to write the query/importer. Yesterday, was all about unit tests for the parser. The next part, will require a test harness[0] (a SwiftUI app). Unit tests won’t be useful for that. It will also be my platform for the ML application. Also, code coverage for the parser tests is only about 85%. I plan to use the test harness to test the remaining 15%.
Many folks would simply rush through this part, but they would end up shipping (or throwing out) the sloppy code they wrote as a lash-up. Even if I decide that I don’t have a current application for ML, I’ll still have a top-Quality SDK, to use as a component for future work.
The app I just released, has a backend server, that I originally designed as a general-purpose application server. It took me seven months to write, as it was written in layers, and each layer was rigorously tested, before moving to the next. The purpose of that project, was as a teaching exercise. I didn’t actually have an application in mind, when I wrote it.
The server is a lot more complex than it needs to be, because it was designed as a general-purpose server, and the current implementation uses a fraction of its capacity. But the server was available, when I started work on the app, and I already knew it was extremely high-Quality, so using it wasn’t a big deal. It probably saved me a couple of months, and I really have a difficult time, trusting backend dependencies.
[0] https://littlegreenviper.com/miscellany/testing-harness-vs-u...
There’s a lot of arguments against that kind of care, as it can be difficult and expensive. There’s a reason that a Mercedes costs a lot more than a Toyota, even though they may seem to be the same car, superficially, but Toyota is probably a more profitable corporation.
It’s just the way I work. I’m not particularly interested in slagging anyone else’s work. I’m just talking about the way I do things.
I think that’s pretty cool, as it means there’s a lot to learn, and I love learning new stuff[0].
[0] https://littlegreenviper.com/miscellany/thats-not-what-ships...
There’s nothing unique about how to handle it. We’ve been doing it for thousands of years, and that’s pretty much exactly what I said. If you want to look at how I do stuff, I posted several articles that I wrote, explaining things in great detail, and here’s my GH page, with links to more detail than you could possibly want[0]. It’s not my responsibility to cut it into bite-size pieces for others.
Complexity is not the problem. In fact, it’s the solution. It’s just how we implement it, that is important, and that’s not new at all. I'm surprised that it is even an issue.
It’s pointless.
Have a great day!
Please don't enable this man when there are literally millions of trained historians you can talk to.
It does seem plausible. We are descended from Cro-magnon man; Neanderthals had bigger brains than us.
https://www.archaeology.org/issues/103-1309/trenches/1138-ne...
And we have smaller brains than our Cro-magnon ancestors.
https://phys.org/news/2011-06-farming-blame-size-brains.html
Doesn't stop him from pushing his own grand narratives of history, apparently.
The claim isn't particularly interesting given the nebulous and culture-bound definition of intelligence, but he's not a person I would tap into for a sense of expertise backed by consensus.
I do like Mac products but just can't get passed their walled garden. However, to have the quality they do I guess it kind of makes sense to have the garden.
I personally like the idea of few-to-none outside dependencies, solid testing and solid, up-to-date documentation. I need to work on being more disciplined and motivated to do it.
I think that most corporations don't like to cede control. I have been told that one of the big reasons for Boeing's troubles, is that they have pushed a lot of important functionality to external contractors.
I vehemently disagree. New codebases might be built on top of large stacks of framework code and infrastructure code, but source repositories ought to actually be tighter and more business-focused than they used to be.
But they're often bloated monstrosities, filled with resume-padding architecture astronaut bullshit, or contorted to fit the latest hip framework's requirements.
Apologies ahead for this long-winded rant, but this exact issue has been boiling my blood for weeks now because I've been wading through the neck-deep mess that a customer has made, such as writing an astonishing three megabytes of source (typed by hand, not generated!) for a single web form. One!
Especially in the era of modern web frameworks, we can avoid most of the "ceremony" of older systems like C or C++ where just making a call to a database was pages of fiddly boilerplate. Instead of embracing this, I see developers reintroducing unnecessary code because they must have gotten so used to the verbosity that they felt exposed without it.
Let me give a concrete example from a codebase I've just spent a week reviewing. I've read through hundreds of pages of code, found dozens of critical resource leaks and security issues, and I'm yet to encounter any actual business logic.
The first mistake that happened was that someone decided to use Angular. Why? It's hip! It's cool! It's what all the kids are doing!
Except now, you have to have an API instead of just web pages. That API will need to serialize things to JSON, so you need to data transfer objects and Newtonsoft code. Suddenly, you need to deal with JWT, and make sure the untrusted client doesn't receive internal (secret) fields from the database. The client is no longer strongly typed or closely tied to the C# code, so you need a full megabyte of test code that you never would have needed before to validate all the Typescript.
This code does nothing of business value. It's a form. You fill it out, and hit submit. There's no "interactivity", there's no single-page-app features used, it's just... waste.
Then, then, on top of that, someone that really likes "grand cathedral" style code architecture decided that this whole thing would be more secure if there was a "middle tier". You know, an API. No, not the Angular API facing the Internet! A back-end API. For no reason. For an app written by two developers that has its own, dedicated database, and will never interact with any other app, tool, or system.
So now you have to write server-to-server API call code including the DTOs and related definitions, the server code, the client code, the service authentication code, the error handling, the distributed tracing, the HTTP header forwarding (securely!), and on, and on.
All typed out by hand, because: "REST is easy!"
No wait, there's more: The original architect also wanted to make the system "database agnostic", so the mid-tier repeats every abstraction. It has interfaces, types, etc... so that you can "swap in" another database. This never happened, and will never happen. This is purely wasted effort.
Then someone put API Management in front of this. Why? Because the word "API" was used. Remember: Angular uses JSON APIs, not HTML, so it's a "thing" that "needs" management. Then that API management -- a cloud SaaS service -- has a reverse proxy in front of it operating in API management mode for "security".
Meanwhile all these layers are riddled with URL injection, SQL injection, and similar basic security errors precisely because it's too hard to add a new field to the form, so developers get lazy and simply add a generic "ExecuteQuery(string sql)" mid-tier method and move on with their lives.
This is why we have "too much code". Because architects can't justify their positions if there's only two icons with a single arrow between them in their designs. Because "hip new thing" is a shiny we all love to chase. Because of magical thinking that assumes that more abstractions means the code is more good.
We have nobody to blame for this but ourselves.
I don’t see any way to avoid it. In fact, every discipline, other than software development, seems to be managing the complexity (albeit, not always well).
My problem is that if each of those layers is bloated to 5-10x the size and complexity they should be, then we have a problem. This can compound, not just additively but multiplicatively. Eventually you end up in a quagmire where nothing works right and you can’t fix anything because that would just reveal more breakage at layers you don’t control.
One thing I noticed over this last week is how much nicer it is to work with something like ASP.NET Core 8.0 instead of the old ASP.NET MVC 5. It’s leaner and meaner, but has enough “batteries included” that I don’t have to drag hundreds of megabytes of slightly incompatible third-party modules in just to get essentials such as JSON and Dependency Injection.
As an industry we need to acknowledge the need for cohesive and consistent platforms, not just components.
We need to occasionally clean house and consolidate, not just keep throwing packages over the fence and calling it a day.
Agreed, but it's also, apparently, a difficult bar to clear.
Apple manages to do it, by having iron-fisted control of the hardware, the operating system, and the SDKs.
But even they are starting to fray. I've been writing Apple code since 1986, and I'm getting familiar with some of the newer stuff, like CreateML and SwiftUI.
There's a ... difference ... in the way that documentation is being handled. In The Days of Yore, we had Inside Macintosh, which was a multi-volume Encyclopedia Britannica-like set of tomes, delving deep into the operating system and SDKs. It was written by really skilled people; often with advanced and esoteric degrees, and with great writing chops.
These days, not so much.
I'm glad for bloggers and online enthusiasts.
Now Microsoft can’t even make a GUI framework that is actually used by anyone, including themselves!
Burned out a few years back writing an admin panel mobile app for an embedded device. The actual panel and the interaction with the device was fine, it was the insanity of the customer architect who only had time to design the system but not build it. You'd think that would have informed them on their excessive creativity, but alas.
One use case was that an operator working with a device could at times need external support. One feature was to be able to dial support from the phone whilst using the app such that the operator could receive instructions, or have support remotely change settings for them.
At least, that's what I thought. That would make sense. But no.
They wanted a full real time socket connection between the hardware, the mobile app, a server, another server and the support app. Important detail is that operators often worked in areas with bad reception.
This meant ensuring that all these connections were up at all times, with error handling, ensuring that messages were delivered and the gist, whilst simultaneously serializing and deserializing these messages and running whatever algorithms over them, often calling other services written in node that would take ages to respond if they didn't just timeout completely (30s!).
Oh, and they wanted to sell this forward so everything, everything, needed to be customizable.
Every meeting with said architect and he had new ideas, new protocols, new solutions.
Eventually I needed to fix a bug in their code, which turned out to be a 15Gig monorepo where 1/3-2/3 of all code was commented out.
You can imagine what the final product was. Which was rolled out to everyone on day one without any testing in live environments.
I'm tired just talking about it.
We stand on the shoulders of giants (or at least geeks with good posture).
One of the things that folks don't like to think about, is that people, these days, aren't really any smarter than they were, five thousand years ago (in fact, Jared Diamond posited that we're actually a bit stupider).
We just have a baseline social infrastructure that allows our "starting point" to be that much more advanced. We are the equivalent of privileged offspring, inheriting family wealth. If we are trained to manage and utilize that wealth, it grows. If we aren't, it will likely get squandered.
I started off, writing my own operating systems in assembly. I even have the manual, for my very first engineering project, in 1987 (including the OS code)[0]. It worked, but it was very primitive. I can't imagine writing an OS in assembler, these days. I'm glad someone else did, in C (or whatever).
So my "starting point," these days, is having a high-quality, parallel operating system, with a developer SDK (again, someone else that knows what they are doing, did the hard work for me), a compiler and linker (thanks, Chris Lattner, et al), etc.
For my own projects, I use a lot of dependencies. The difference is, is that I write most of my own dependencies. These are standalone modules, with each one having its own project lifecycle, documentation, testing, etc. Even if no one else will use them, I still publish them on GitHub, as supported open-source projects, because that means that I don't need to worry about them, when I use them in my own aggregate projects.
Basically, "divide and conquer." Big things are composed of small things. It's important for me to do a good job on the small things, so the big things turn out better.
But that's just me, and the way I work. YMMV.
[0] https://littlegreenviper.com/wp-content/uploads/2022/07/TF30... (Downloads a PDF)