Intermediate developers, bitten by their past mistakes, decide to future proof their work. But they don’t just look one or two steps ahead, rather they try to look five steps ahead and identity problems that do not and may never exist. Consequently, they over-engineer, over-abstract, and over-complicate everything. Carmack’s recent “But here we are” speech resonated with me.
Advanced developers identify the right balance for each task between simple and complex, concrete and abstract, and pragmatic and idealistic. In my experience, they favor simple and pragmatic solutions, use abstraction effectively, and can satisfy near-term goals quickly without painting themselves into a corner. “As simple as possible, but not simpler.”
I try to avoid working for tech leads stuck in the second phase, which is not uncommon. If you suggest taking the “ifs and for loops” approach of solving a simple problem with a simple solution, they’ll assume you’re on the wrong side of the bell curve.
Debated for a long time whether that methodology was stuck in the second phase or if it was actually the third. Still don't have an answer, but these days I think having a plan is better than just letting engineers run roughshod, as long as the conventions are easy to follow.
In the first kind, all relevant developers have deep expertise in the system, or build towards having deep expertise. There's an expectation that flexible abstractions will be used, not abused, unless it's one of those scenarios where the use outweighs the abuse. The abstractions are tomato cages, and they're there to support the system as it grows, provide some structure, but not to strangle it.
In the second kind, the default expectation is that a developer will have little to no familiarity with the system, they will be isolated from it as much as possible, and they will be given such a tightly constrained sandbox that they can't break anything outside it. You will write your little plugin, or whatever, get in and out, and you're done.
These can both be useful kinds of systems/codebases in orgs of any size. The first kind of codebase can enable an experienced team to move really fast and be extremely productive. The second kind of system can help lots of different teams of different skill levels jump in and leverage your system with little required knowledge, and thus be productive that way. So there's really no way to say one of these patterns is good or bad.
But in general if you churn in and out a bunch of replaceable cog code monkeys, probably low-paid, the second kind of system just ends up working better. Giant "enterprise" software shops like parent poster aluded to typically end up in this kind of high turnover scenario after enough finance/MBA people have been brought in, hence their bad rap.
On the other hand, on the right conditions, the amount you learn on a good team is ridiculous compared to what you’d do alone. Weeks vs years kind of thing.
A: The novice thinks twice before doing something stupid.
So juniors thinks twice before doing stupid simple stuff. Intermediates thinks twice and does smart stuff. Seniors only does smart stuff where it is needed and does the stupid simple stuff without thinking in most places.
That said for that specific meaning I would prefer something like - The novice fears simplicity.
Maybe throw in a - The expert loves it - at the end.
Instead, this thing would give the most "enterprisey" Spring JEE application a run for its money with its endless annotations, dependency injection magic, all sorts of pseudo-types - both the "built-in" Python 3 ones like Set and List, but also the libraries like Pydantic. But unlike Java, these types aren't even really guaranteed by the language at compile time, so even if your IDE plugin can successfully detect them, things will still (silently) slip through at runtime.
The async functionality that's been bolted on to the language is worse than even the old and confusing Java multi-threading primitives, and the funny thing is it still doesn't actually run things in multiple threads. For that, your simple Rest API is running on layers of C programs like Uvicorn which itself is then wrapped by another few processes running Gunicorn which in turn is probably running behind NGINX. LOL, and we thought the Servlet stuff with Tomcat and JBoss was clunky - this is insane.
To be honest, if there ever was a sweet spot for Python, it would have been for smaller code bases that weren't complex enough for big "enterprisey" langs like .Net or Java, but were more permanent and complex than shell scripts or (back in the day) Perl could handle.
But nowadays, I don't think modern Python fits any use case real well. It's still dynamically typed, slow, single-threaded, and has a poorly managed ecosystem and hodge-podge of tools like virtualenv, pyenv, poetry, etc. that never quite become standardized and stable.
So unless you've got a bunch of Python experts who aren't interested in moving to a better lang, I'd find it hard to go with Python for new projects.
This sounds like what happens when a bunch of Java/C# developers jump over to python without learning the "python way" - this is more related to the developers than the project
> But nowadays, I don't think modern Python fits any use case real well
Python has effectively taken over the data science / machine learning space. For most use cases, the algorithms are massively more important than the language.
> poorly managed ecosystem and hodge-podge of tools like virtualenv, pyenv, poetry, etc. that never quite become standardized and
This is true, but Java and C# also have many issues in this respect. The move from Java 8->11 is particularly painful - many fundamental libraries related to security or connection handling were not backwards compatible. Many libraries now require multiple branches for different JDK levels. Maven and Nuget are about as good as pip - they all have weird edge cases.
I use both Java and Python on a daily basis - each has their strengths and weaknesses. Java is great if need long running processes with weeks/months of uptime, Python is great for backend data manipulation and analysis.
https://motherfuckingwebsite.com/
Where does this fall?
The silly thing is that it's literally one CSS property, max-width, that takes care of that problem.
Everything is as simple as it can be, no simpler and no more complex. Sometimes a bunch of flat JSON files in an S3 bucket is enough of a database, you don't need a 42 machine Aurora cluster.
All fancy "Machine learning" stuff really is just a bunch of ifs and for loops internally :D
Watching this right now and all I can think about is Microsoft Bob.
I have no beef with if+for, but a large part of the reason they're "goto tools", if you will, is because industry is slow to assimilate many state-of-the-art ideas, sometimes by as much as 40 years.
Simpler building blocks does not necessarily mean simpler solution. If only!
This comment sure indicates to me where you most likely are on the curve.
In all seriousness, I think this is considerably off the mark. After enough experience you realize that expressivity and convenience are antipatterns and don't actually simplify things but are harbingers of complexity, bugs, tech debt, even the downfall of organizations and products.
Saying it is all ifs and for-loops is completely true. Everything else, all the abstractions and high level features, are just sugar.
I try to keep a healthy and balanced diet, myself.
> industry is slow to assimilate most state-of-the-art ideas, sometimes by as much as 40 years.
Most of those ideas are terrible. The industry is so incredibly young and has experienced so much change over those 40 years that I have a hard time accepting the notion that the industry is slow to adopt. The reason the old building blocks are still popular is because they are a thin abstraction over how computers work, and ultimately that is at the root of everything we do.
If+for have no deeper foundational significance in the construction of programs or computations, literally, than say a lambda function. But because the latter is unfamiliar, it's spoken about in the same manner you present: as if it is some highly abstract, complicating, high-level feature (when truly that take is just baloney).
Because the "expressive abstractions" are much easier to reason about and save programmers lots of mental effort. And, as I have commented upthread, ifs and for loops are by no means the only such abstractions.
> because the latter is unfamiliar, it's spoken about in the same manner you present: as if it is some highly abstract, complicating, high-level feature
If expressing your program in the lambda calculus is easy for you to reason about and saves you enough mental effort, go for it. But don't expect your code to be readable or understandable by many other people. The reason why ifs and for loops (and other "expressive abstractions", since as I have said, those are by no means the only ones) are ubiquitous in programs is that they are easy for lots of programmers to reason about. Whereas the lambda calculus is only easy for a very small subset of programmers to reason about.
We now have a chicken-egg problem. I can freely admit that for+if is easy for programmers to understand solely because of how we are educated, and not due to any Bruce Lee hocus pocus about simplicity or fundamentalism, as so many others here suggest.
A programmer who, say, learned from SICP first would find a for loop awkward and bizarre when you could "just" tail-recurse.
Sure, functions and their composition are very often another highly useful abstraction that is easy to reason about and saves the programmer a lot of mental effort.
> We now have a chicken-egg problem.
No, just a recognition, which was not there in the original claim, that ifs and for loops are not the only useful abstractions.
> A programmer who, say, learned from SICP first would find a for loop awkward and bizarre when you could "just" tail-recurse.
Perhaps, but I think such a programmer who needed his code to be readable and understandable by a lot of other people in a large project would end up inventing a "for-loop" macro or something similar that, while it might use tail recursion under the hood, would express the construct in a way that programmers could understand regardless of how they learned programming.
But symbol calculus is a highly abstract, complicating, high-level system assembled out more reality-based systems beneath it. If it seems simple to you, you're just under the curse of knowledge.
And I don't know what a "reality-based system" is.
I think the reason that stateful, imperative code is the default is because that is how we tend to "think". Example, if I wanted to get a list of items for the grocery store, I get a piece of paper (instantiate an array), go through the list of things I might need (loop through a source array) and write down the ones I definitely need (append onto the destination array). If I run out of space, I get a new piece of paper and continue the process.
With that algorithm, I can stop and test each individual statement; I don't need to "think about it" because I can write and execute a tiny part of it (even if I'm only executing it in my mind).
In a functional style, I have to think more declaratively. Given all my items, filter out the ones that need replacing and give me that list. It's much easier to reason about abstractly, but I have no details about w how that's actually happening.
I think this kind of thinking is superior most of the time: I would rather read map.filter.reduce more than a for loop with branch statements any day of the week, but I am trusting the implementations of the mapping and filtering and reducing functions. Of course, for any non-trivial algorithm one is still passing a function into the map/filter/reduce anyway so I can still reason about those smaller stateful sections without worrying about the map/filter/reduce piping.
Perhaps I've never worked in a truly pure functional style so I may not be dealing with the mountains of abstractions others seem to complain about
It’s not like the machine code will look much closer to your C code either. That’s also a spell of “compiler writers and hardware vendors trying to uphold the view that C programmers are so close to hardware and that memory access is flat”.
In languages and libraries that allow FSM and pure functional kernel based designs you can get just as clear logic that is expressible not just to the programmer but also to business personnel. It's counter-intuitive to a certain extent because so much of programming is built around imperative programming but FSM based logic is and will continue to be easier to understand long term because you can trivially visualise it graphically. This ultimately is what a lot of the functional paradigm is built around. Use the mathematical and graphical representations we've used to understand systems for decades. They are well understood and most people can understand them with little to no additional education past what they learned in their business or engineering bachelors degrees.
Most professions have a healthy respect for the base materials they work with no matter how high the abstractions and structures they build with it go. Artists know their paints, stone, metal, etc. Engineers know their melaterials as well. They build by taking the advantages of each material into consideration, not assuming that it's no longer relevant to their job because they get to just work in I-beams. Programmers would do well to adopt a healthy respect for their base materials, and it seems like often we don't.
I disagree with how your use of "just" here. It's common for programmers to dismiss the importance of syntax but syntax and notation are the interface and UX between the language semantics and your brain. It's no less important to get this right. There's a half-joke that Europe was able to rapidly advance in Calculus beyond Britain due to the superiority of Leibniz notation.
> healthy respect for their base materials
What's unique about computers is the theoretical guarantee that the base does not matter. Whether by lambda calculus, register machines or swarms of soldier crabs running from birds in specially designed enclosures, we're fine as long as we appropriately encode our instructions.
> bunch of branches and loops
You could also easily say it's just a bunch of state machines. We outsource tedious complexity and fine details to compiler abstractions. They track things for us that have analogues in logical deduction so that as long we follow their restrictions, we get a few guarantees. When say, writing asynchronous non-deterministic distributed programs, you'll need all the help you can get.
Even designing close to the machine (which most programs will not need) by paying attention to cache use, memory layout, branch divergence or using SIMD remain within the realm of abstractions.
This is a contradiction. Simplicity is obtained through abstractions. As an example, fifty years ago, 'goto' reigned supreme. Then structured 'if/else/for' took over, as the superior abstraction over 'goto'. Now use of 'goto', while being far simpler to implement and closer to the bare metal, is commonly derided by many programmers.
The long term trend in software is constantly increasing abstraction hiding mountains of complexity, which increases both simplicity and complexity. Writing print('hello world') is simple, but the compiler, OS, silicon, etc. that makes that simplicity possible is extremely complex.
It is a must, because decades of business requirement built on top each other without understanding the whole is complex. Writing a JIT-compiler that can routinely change between interpreting code and executing it natively, a database optimizing queries, a mathematical library using some fancy algorithm are all complex, in a way that is not reducible.
Complexity easily outgrowth even the whole of our mathematics, we can’t prove any non-trivial property of a program, halting problem, etc.
So all in all, no, we can only respect our “base materials” by finding the proper abstraction for the problem, as our base material is complexity itself. It might be for loops and ifs, but it very well be a DSL built on top of who knows how many layers, because at that abstraction level can we even start to map the problem domain to human consumable ideas.
In my experience programming programming with primitives and basic flow control operations frequently tends to be at least be order of magnitude faster than more complex state management paradigms. Compilers are very good at optimizing that style of code. Loops often get unrolled, the and the branch predictor is kept happy. A good compiler may use vector expressions.
In many cases with cold code it flat out doesn't matter, the CPU is plenty fast, but when it does matter, explicit if-and-for code absolutely mops the floor with the alternatives.
You can optimize one specific query (quite painstakingly, I believe) to beat a general db, but it is very far from clear that “for loops will beat higher level abstractions”, imo.
With my own set-up I can have my data in pre-calculated immutable tables that are memory mapped off disk. That means I can write lock-free code. My updates are large hour-long batch operations off a written journal.
For the lexicon I'm also rolling a special one-way compression scheme/hash function that guarantees uniqueness but only works due to quirks with the data.
General purpose databases can't do these things, because then they wouldn't be general purpose databases.
SQL databases perform really well on average given your query is relatively optimized, but they do not represent the pinnacle of performance. Any time you want the fastest most efficient code to deal with a particular use case, it pays enormous dividends to roll your own implementation that has domain awareness. It means you can cut corners general solutions simply can't.
I’m yet to see a secretary who could “return a new table state such that as if documents became a right fold as binding together a map of composition of signing and copy routines over documents” instead of “get these documents from the table, copy, sign and put them back in a binder”. This is a nonsense some of us want to believe in, but it is not true.
If the project had instead been designed to have less unnecessary state and “transitions” it would have been a lot easier to make changes.
All those ideas sound good by themselves but they are really bad for “defensive” coding. Good luck selling a project to refactor something when it’s going to take multiple people-years. Once you’ve made the mistake of committing to an inflexible design it’s either that, replace/ignore the thing, or deal with low productivity as long as the thing exists.
so was the chosen model the issue or choosing a state machine model at all?
Imperative code:
Take A and B.
Add C ml of Water.
Stir for 2 minutes.
If it thickened, add D grams of flour, else go back to stiring.
Imperative code quickly devolves to a point where it is no longer understandable by anyone but the developers who wrote it.
For problems where those are the right tools, sure. But they aren't the right tools for all problems any more than ifs and for loops are.
You could just as well say that ifs and for loops are just sugar for gotos and all programming is just gotos.
The reason ifs and for loops are used instead of gotos is that they are very useful abstractions that are easy to reason about and save the programmer lots of mental effort. But they are not the only such abstractions.
To the extent that other abstractions can create problems, it's not because they're really just sugar for ifs and for loops, it's because they are not well crafted abstractions so they are not easy to reason about and don't really save the programmer any mental effort. But there are plenty of abstractions other than ifs and for loops that are well crafted and do save the programmer mental effort, in many cases lots of it.
Suggesting that experience leads to jettisoning expressivity is at odds with my direct observations of experienced software engineers working in large teams. The more experience, the _better_ the engineer gets at picking the right level of abstraction to write code that can be maintained by others. Picking a single point on the abstraction spectrum (just above goto but not below it!) is far too rigid for the diversity of tasks that software engineers need to solve.
However the code abstractions people used 50 years ago could still implement all programs we have today. We added a lot of sugar on top, but the core of coding remained the same. For example, you could implement Map, Reduce etc in C, they are just functions and not novel features of programming.
Obviously, it can only be read by someone who can also understand programming beyond ifs and fors. That’s a non-starter in most environments - enterprise or otherwise.
Funny enough, I see most innovations coming from consultants who do the same work for multiple companies and realise the repeating patterns and extract an abstraction.
I wouldn't say that they are the standard now, but using and mastering all features in a language is hard.
Add to that design patterns, classes and code layout it becomes a full-time job to keep up.
I have been in contact with code most of my professional life, but still isn't comfortable writing large amounts of code. The simple reason is that i don't do it full-time.
Here are the features in C# just to illustrate how complex a programming language is.
However, most of those features are relatively standard and are more conceptual than syntactical in nature. Bashing people because they don't know stuff is stupid and counterproductive, but I shouldn't be forced to code in the programming equivalent of roman numerals just because someone else can't be properly fucked to understand lambdas or interfaces or generics, all stuff that's literally covered in your run-of-the-mill CS 101 course.
It all boils down to having enough humility and empathy to understand that other people are not, in fact, the same as us.
For assimilation to happen, the state-of-the-art solution also has to result in a net gain over the existing solution, and the higher the differential in complexity between the two, the bigger that gain has to be.
There's no "rubber meets the road" in OPs position because there's no cost in their calculations.
“If” however seems pretty fundamental.
With at additional level of abstraction you could say “goto jumps”, but “if and loops” gives an commonly understandable logic for everyone; deeper abstractions increase reading complexities, while higher abstraction is achieved via functions and overall architecture.
Scaling up those “if and loops” is the challenge as a team or a single, with the common goals being to keep the software under control.
I see what you did there.
Isn't this just a complicated case of ifs and fors?
For example, how do you handle authorization, logging, and how do you make the code maintainable? That's a really tough problem that requires a lot of thought about the overall system design.
And of course it's always a lie to say that performance and resource usage aren't a concern -- they're not a concern until they are.
How convenient that the software industry is about 40 years old. So these ideas should "break through" this invisible arbitrary corporate waiting area into the limelight any day now, right?
- you don't need to write classes for everything
- it's okay to write something in a more verbose way to make it clear for other people
- your tools don't need to be perfect to get things done
I need more of these, maybe some that aren't as reductionist as Carmacks's original post.
https://tenderlovemaking.com/2016/02/05/i-am-a-puts-debugger...
In rare cases I pull out a real debugger, but most of the time the right prints in the right places are just as good. I can also iterate much faster because I'm not jumping between the code the the debugger, or pulling the debugger out of the loop it's stuck in.
“Before I learned the art, a punch was just a punch, and a kick, just a kick. After I learned the art, a punch was no longer a punch, a kick, no longer a kick. Now that I understand the art, a punch is just a punch and a kick is just a kick.”
not my favorite source since it doesn't go into the 'scaling the mountain' bit, but every source that talks abt that part seems to be...eh: https://buddhism.stackexchange.com/questions/15921/what-is-t...
Q: What is the difference between an enlightened person and an ordinary person?
A: There is no difference, but only the enlightened person knows this.
After agile: it's not done yet
We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.
T. S. Eliot - Little Gidding
Like them, I left a settled life, I threw it all away
To seek a Northwest Passage at the call of many men
To find there but the road back home again
Stan Rogers, "Northwest Passage"
Anyone who has ever written any software has felt like the unenlighted half-person in the middle of that distribution at least once -- for example, when learning how to code with a different approach in a new language.
I have felt like that more than once. Everyone here has, I suspect.
--
[a] https://twitter.com/nice_byte/status/1466940940229046273/pho...
but you hear those $fancy_principles / fp / hard oop / "clean code" evangelists, and then you go to any repo of real world software - linux, compilers, kubernetes, git, blablabla and everywhere you see for loops, goto, ladders of if statements
Also, let’s be honest, C does all these “virtual method” magic on a per-project basis which will not be understood by any tool ever (all those function pointers to whole new implementations passed from God knows who, with barely any typing). At least FP and OOP knowledge somewhat transfers and is queryable by tools.
My claim is:
Almost all real world, OSS, big, battle-proven code bases are nowhere even close to evangelist's "sanity".
They're full of loops, nulls, gotos, ifs and all ""bad stuff""
>all the web servers running the whole cloud
I literally opened first file that I've seen in nginx repo and take a look at this:
shitton of ifs, for loops gotos and all of that in one, a few hundreds lines long method.
https://github.com/nginx/nginx/blob/master/src/http/ngx_http...
>On the other hand compilers are often written in C++
are they?
>or are bootstrapped in the very same language though.
thus what?
"...there's way too much information to decode the Matrix. You get used to it, though. Your brain does the translating. I don't even see the code."
I describe this path of discovery as:
beginner: function over form
intermediate: form over function
transcendence: form is function
However, I will disagree that coding is just about ifs and for loops. To me, coding, programming, software development, or whatever you want to call it is about three things: how to use a computer to do something, communication between people (including your future self), and how to think about a domain. “ifs and for loops” does not capture this.
https://twitter.com/nice_byte/status/1466940940229046273
The more I do this, the more I gravitate towards the simple things.