so much this. it is _much_ easier to refactor copy pasta code, than to entangle a mess of "clean code abstractions" for things that isn't even needed _once_. Premature Abstraction is the biggest problem in my eyes.
Write Code. Mostly functions. Not too much.
The interesting thing about DRY is that opinions about it seem to depend on what project you’ve worked on most recently: I inherited a codebase written by people skeptical of DRY, and we had a lot of bugs that resulted from “essential duplication”. Other people inherit code written by “architecture astronauts”, and assume that the grass is greener on the WET side of the fence.
Personally, having been in both situations, I’d almost always prefer to untangle a bad abstraction rather than maintain a WET codebase.
Or, put another way: I'd much rather deal with duplication than with coupling problems.
I think the single biggest factor determining the difficulty of a code change is the size of the codebase. Codebases with a lot of duplication are larger, and the scale itself makes them harder to deal with. You may not even realize when duplication exists because it may be spread throughout the code. It may be difficult to tell which code is a duplicate, which is different for arbitrary reasons, and which is different in subtle but important ways.
Once you get to a huge sprawl of code that has a ton of mostly-pointless repetition, it is a nightmare to tame it. I would much rather be handed a smaller but more intertwined codebase that only says something once.
Often, they are.
IME, a very common pattern is divergent copypasta where – because there is no connection once the copying occurs – a bug is noticed and fixed in one place and not in others, later noticed separately and fixed a slighly different way in some of the others, in still others a different thing that needs done in the same function gets inserted in between parts of the copypasta, etc. IT’s still essentially duplication – more over its still the same logical function being performed different places, but in slightly different ways, creating a singificant multiple on maintainance cost, which – not literal code duplication in and of itself, is the actual problem addressed with DRY, which is explicitly not about duplication of code but single source of truth of knowledge: “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system”. Divergent implementations of the same logical function are different representations of the same knowledge.
Repeating what others already mentioned, often it can be the same thing but written in a slightly different way. Even basic stuff like string formatting vs string concatenation can make it non-obvious that two pieces of code are copies.
Fixing a bad abstraction is only hard because there's some weird culture about not tearing down abstractions. Rip them apart and toss them on the heap. It's a million times easier than finding duplicate code that has inadvertently drifted apart over time.
When I asked exactly what they were trying to accomplish they kept saying "they didn't want caching in two places". I think anyone with experience can see that these items are unrelated beyond both having the word "cache"
This is what was actually hanging them up ... the language. Particularly the word "cache". I've seen this trap walked into over and over. Where a person tries to "unify" logic simply because two sets of logic contain some of the same words.
So long as it remains identical. Refactoring almost identical code requires lots of extremely detailed staring to determine whether or not two things are subtly different. Especially if you don't have good test coverage to start with.
Yeah, nah. There are diffing tools.
Just wanted to appreciate the nod to good nutritional advice here ("Eat food. Mostly plants. Not too much"), well done
DRY and WET are terms often used as objective measures of implementations, but that doesn't mean that they are rock solid foundations. What does it mean for something to be "repeated"? Without claiming to have TheGreatAnswer™, some things come to mind.
Chaining methods can be very expressive, easy to follow and maintain. They also lead to a lot of repetition. In an effort to be "DRY", some might embark on a misguided effort to combine them. Maybe start replacing
`map(x => x).reduce(y, z => v)`
`mapReduce(x => x, (y,z) => v)`
But there may equally be situations where consolidation makes sense. For example, if we're in an ORM helper class and we're always querying the database for an object like so
`objectContext.Orders.Select(e => e.id = y).Include(e => e.Customers).Include(e => e.Bills).Include(e => e.AwesomeDogs)...`
`orderIncludingCustomersBillsAndDogs(id) => ...`
Don't needlessly copy-pastes that which is abstractable.
Don't over abstract at the cost of simplicity and flexibility.
Don't be a zealot.
I totally agree assuming that there will be time to get to the second pass of the "write everything twice" approach...some of my least favorite refactoring work has been on older code that was liberally copy-pasted by well-intentioned developers expecting a chance to come back through later but who never get the chance. All too often the winds of corporate decision making will change and send attention elsewhere at the wrong moment, and all those copy pasted bits will slowly but surely drift apart as unfamiliar new developers come through making small tweaks.
When we got absorbed by a larger team and were going to have to "merge" our code review / git history into a larger org's repos, we learned that a sister team had gotten in big trouble with the language cops in HR when they discovered similar language in their git commit history. This brings back memories of my team panicked over trying to rewrite a huge amount of git history and code review stuff to sanitize our language before we were caught too.
To me they're harmless jokes but maybe someone will point out they're ableist?
The biggest thing is that we used non-PC language like "retarded" very casually, not to mention casually swearing in commit messages e.g. "un-fucking up my prior commit". Our sister team got in trouble for "swears in the git commit history", so we wanted to get ahead of that if possible.
In a healthy company culture, we'd just say "okay we'll stop using these terms", but the effort was made to erase their existence because this was a company where non-engineering people (e.g. how well managers and HR liked you) was a big factor in getting promoted.
Once I realized how messed up that whole situation was, I left as fast as I could.
Its easy to refactor if its nondivergent copypasta and you do it everywhere it is used not later than the third iteration.
If the refactoring gets delayed, the code diverges because different bugs are noticed and fixed (or thr same bug is noticed and fixed different ways) in different iterations, and there are dozens of instances across the code base (possibly in different projects because it was copypastad across projects rather than refactored into a reusable library), the code has in many cases gotten intermixed with code addressing other concerns...
Think about data structures (types) first. Mostly immutable structures. Then add your functions working on those structures. Not too many.
"Bad programmers worry about the code. Good programmers worry about data structures and their relationships." - Linus Torvalds
He wasn't kidding!
And the bit about "immutable structures". I doubted for infinity-number-of-years ("oh, waste of memory/malloc!"). Then suddenly your code needs to be multi-threaded. Now, immutable structures looks genius!
Good code should be written to be easy to delete.
'Clever' abstractions work against this. We should be less precious about our code and realise it will probably need to change beyond all recognition multiple times. Code should do things simply so the consequences of deleting it are immediately obvious. I think your recommendations fit with this.
This article tends to make the rounds on here every once in a while: https://programmingisterrible.com/post/139222674273/how-to-w...
Etc... etc...
(Not saying that this is the only quality of good code or that you won’t have to trade some of the above for performance or whatnot at times).
There are two opposite situations. One is when several things are viewed as one thing while they're actually different (too much abstraction), and another, where a thing is viewed as different things, when it's actually a single one (when code is just copied over).
In my experience, the best way to solve this is to better analyse and understand the requirements. Do these two pieces of code look the same because they actually mean thing in the meaning of the product? Or they just happen to look the same at this particular moment in time, and can continue to develop in completely different directions as the product grows?
In practice (time pressure) you might end up duplicating it many times, at which point it becomes difficult to refactor.
It's a similar design and planning principle to building sidewalks. You have buildings but you don't know exactly the best paths between everything and how to correctly path things out. You can come up with your own design but people will end up ignoring them if they don't fit their needs. Ultimately, you put some obvious direct connection side walks and then wait to see the paths people take. You've now established where you need connections and how they need to be formed.
I do a lot of prototyping work and if I had to sit down and think out a clean abstraction everytime I wanted to get for a functional prototype, I'd never have a functional prototype--plus I'd waste a lot of cognitive capacity on an abstraction instead of solving the problem my code is addressing. It's best, from my experience, to save that time and write messy code but tuck in budget to refactor later (the key is you have to actually refactor later not just say you will).
Once you've built your prototype, iterated on it several times had people break designs forcing hacked out solutions, and now have something you don't touch often, you usually know what most the product/service needs to look like. You then abstract that out and get 80-90% of what you need if there's real demand.
The expanded features beyond that can be costly if they require significant redesign but at that point, you hopefully have a stable enough product it can warrant continued investment to refactor. If it doesn't, you saved yourself a lot of time and energy worrying trying to create a good abstract design that tends to fail multiple times at early stages. There's a balance point of knowing when to build technical debt, when to pay it off, and when to nullify it.
Again, the critical trick is you have to actually pay off the tech debt if that time comes. The product investor can't look bright eyed and linearly extrapolate progress so far thinking they saved a boatload of capital, they have to understand shortcuts were taken and the rationale was to fix them if serious money came along or chuck them in the bin if not.
This is overstated. Not all abstractions are obvious substitutions. To elaborate: languages vary in their syntax, typing, and scope. So what might be an 'easy' substitution and one language might not be easy in another.
That's still too much of a "rule".
Whenever I feel (or know) two functions are similar, the factors that determine if I should merge them:
- I see significant benefit too doing so, usually the benefit of a utility that saves writing the same thing in the future, or debugging the same/similar code repeatedly.
- How likely the code is to diverge. Sometimes I just mark things for de-duping, but leave it around a while to see if one of the functions change.
- The function is big enough it cannot just be in-lined where it is called, and the benefit of de-duplication is not outweighed by added complexity to the call stack.
Document.
Your.
Shit.
everything else can do one. Just fucking write documentation as if you're the poor bastard trying to maintain this code with no context or time.
Comments in code can lie (they're not functional); can be misplaced (in most languages, they're not attached to the code they document in any enforced way); are most-frequently used to describe things that wouldn't require documenting if they were just named properly; are often little more than noise. Code comments should be exceedingly-rare, and only used to describe exception situations or logic that can't be made more clear through the use of better identifiers or better-composed functions.
External documentation is usually out-of-sight, out-of-mind. Over time, it diverges from reality, to the point that it's usually misleading or wrong. It's not visible in the code (and this isn't an argument in favor of in-code comments). Maintaining it is a burden. There's no agreed-upon standard for how to present or navigate it.
The best way to document things is to name identifiers well, write functions that are well-composed and small enough to understand, stick to single-responsibility principles.
API documentation is important and valuable, especially when your IDE can provide it readily at the point of use. Whenever possible, it should be part of the source code in a formal way, using annotations or other mechanisms tied to the code it describes. I wish more languages would formally include annotation mechanisms for this specific use case.
yes, and the solution is to actually document.
> wouldn't require documenting if they were just named properly
I mean not really. Having decent names for things tells us what you are doing, but not why.
> only used to describe exception situations
Again, just imagine if that wasn't the case. Imagine people had the empathy to actually do a professional job?
> The best way to document things is to name identifiers well, write functions that are well-composed and small enough to understand, stick to single-responsibility principles.
No, thats the best way to code.
The best way to document is to imagine you are a new developer to the code base, and any information they should know is where they need it. Like your code, you need to test your documentation.
I know you don't _like_ documenting, but thats not the point. Its about being professional, just imagine if an engineer didn't _like_ doing certification of their stuff? they'd loose their license. Sure you could work it out later, but thats not the point. You are paid to be professional, not a magician.
On the contrary, I like writing documentation. What I can't stand is unnecessary documentation; documentation of obvious things; documentation as a replacement for self-documenting code; documentation for a vague ill-defined future use-case that probably won't exist (YAGNI); and many of the other documentation failures I've already mentioned. It has nothing to do with my professionalism. It has to do with wasting time and energy and using it as a crutch in place of more valuable knowledge transfer mechanisms.
Typings, well-named functions and variables, consistent APIs and once in a while, a comment to summarize rationales and point out edge cases, and a README file to explain the setup and entry points.
That is generally all you need in terms of documentation. I've dealt with plenty of codebases like these and they're a delight to learn.
Boto3, its massive. The only reason its tolerable is that its got a large amount of accurate documentation. On the face of it, its an utter arse, because the "client" section of the library is deeply unpythonic (not in the oh its not pretty sense, as it its barely OO and uses capitals for argument names)
If that was "self documenting" I wouldn't be using it.
Take flask for example, its usable because its got good documentation and its well thought out.
but, as I keep on having to tell people. Good function names and tying is not documentation. These tell you what but not why. You don't need documentation if you are working on the same codebase day in day out. But if you have to pick it up again after 6months, you'll wish that you had properly documented it.
I get that you are proud that you don't need documentation. I don't need documentation, I want it because it makes my life 500% easier.
I don't think you understand what my post is trying to convey. Documentation isn't a sin, it's great when necessary. In the case of boto, it's obviously necessary. In the case of Flask (a framework where you're not going to read the code), it's obviously necessary.
You'll notice, if you read through the Flask codebase, that it's only documented in terms of user-facing functions, which end up in the official end-user documentation. In that sense, the codebase itself is in fact mostly self-documenting.
There's a difference between end-user documentation and developer-facing documentation.
One of the most difficult to argue comments in code reviews: “let’s make it generic in case we need it some place else”. First of all, chances that we need it some place else aren’t exactly high, unless you are writing a library and code explicitly design to be shared. And even if such need arises, chances of getting it right generalizing from one example are slim.
Regarding the book though, I have participated in one of the workshops with the author and he seemed to be in favor of WER and against “architecting” levels of abstraction before having concrete examples.
You can disagree over what exactly is clean code. But you will learn to distinguish what dirty code is when you try to maintain it.
As a person that has had to maintain dirty code over the years, hearing someone saying dirty code doesn't exist is really frustrating. Noone wants to clean up your code, but doing it is better than allowing the code to become unmaintainable, that's why people bring up that book. If you do not care about what clean code is, stop making life difficult for people that do.
Not sure why you're being downvoted, but as an unrelated aside, the quote you're responding to literally did not say this.
> that's why people bring up that book
I think the point is that following that book does not really lead to Clean Code.
And that is why I started saying "You can disagree over what exactly is clean code". Different projects have different requirements. Working on some casual website is not the same as working on a pacemaker firmware.
I'd say if you want good code in your working environment, set up a process for requiring it. Automated testing and deployment, and code reviews would be the way to go. Reading Clean Code won't get you there.
> As a person that has to maintain dirty code
This is a strange credential to present and then use as a basis to be offended. Are you saying that you have dirty code and have to keep it dirty?
Let's say you are a feeding a dog. You can estimate what amount of food is dog too little, and what amount of dog food is too much... but now, some jerk comes around and tells you they're going to feed the dog the next time. You agree.
You check the plate, and there's very little food in it. So you say: "hey, you should add more dog food".
Then, the jerk reacts by putting an excessive amount of food in it, just to fuck with you. Then you say "that's too much food!"... So then the jerk reacts saying "you should tell me exactly how many dog pellets I should put on the plate".
Have you ever had to count dog pellets individually to feed a dog? no. You haven't, you have never done it, yet you have fed dogs for years without problems just using a good enough estimate of how much a dog can eat.
Just to please the fucking jerk, you take the approximate amount dog food you regularly feed the dog every day, count the pellets, and say: "there, you fuck, 153 dog pellets".
But the jerk is not happy yet. Just to really fuck with you, the guy will challenge you and say: "so what happens if I feed the dog 152 pellets, or 154... see? you are full of shit". Then you have to explain the jerk that 153 was never important, what's important is the approximate amount of dog food. But the jerk doesn't think that way, the jerk wants a fucking exact number so they can keep fighting you...
Then the jerk will probably say that a dog pellet is not a proper unit of mass, and then the jerk will say that nutrients are not equally distributed in dog pellets, and the bullshit will go on and on and on.
And if you are ever done discussing the optimal number of pellets then there will be another discussion about the frequency in which you feed the dog, and you will probably end up talking about spacetime and atomic clocks and the NTP protocol and relativity and inertial frames just to please the jerk whose objective is just to waste your time until you give up trying to enforce good practices.
And this is how the anti-maintainability jerks operate, by getting into endless debates about how an objective measure of things is required, when in reality, it's not fucking needed and it never was. Estimation is fine.
Just like you won't feed a dog a fucking barrel of dog food you won't create a function that is 5 thousand lines of code long because it's equally nonsensical.
So in the end, what do you do? you say: this many lines of code in a function is too fucking much, don't write functions this long. Why? because I say so, end of debate, fuck it.
There's a formal concept for this in philosophy, but that's up to you to figure out.
I think I know this one! Is it the paradox of the heap?
The cost of analysis outweighs the benefit.
In this case the cost of counting food pellets versus the benefit of precision; but in software generally it is the high cost of (accurate) estimating versus the benefit of just getting on with it.
It is also, to some degree, the impossibility of estimating given that the task is given to someone who is an expert at writing code, not estimating. Coding expertise give insight into how long coding tasks take, but that it the least critical component of estimating a task. Writing code and seeing if it works is the best estimate; sometimes the prototype works first time, never always.
best example of this: https://xkcd.com/1445/
That's probably mistake number one right there.
There's another formal concept for this in philosophy, related to wrestling with pigs.
Despite approaching this from completely different angles I think we're roughly pointing in the same direction.
A pragmatist would allow for a strong baseline of good practices and automated rules, with some room for discretion where it counts. That way, no one gets bogged down in exhausting debates, and people who work better with strict constraints can avoid ambiguity, but you can still bend the rules with some justification. I don't like all of the rules, but it's easy to fall back on the tooling than it is to hash out another method length debate.
As far as maintaining code goes, I've had the (mis)fortune of dealing with overly abstracted messes and un-architected spaghetti. I'm not sure which is worse, but what I rarely have to deal with now is all the crap that is auto-formatted away.
If I'm involved in any debate, it's around architecture and figuring out how to write better code by properly understanding what we're trying to achieve, rather than trying to lay down Clean Code style design patterns as if they were Rubocop rules.
Excessive abstractions, leaky abstractions and such are indeed a problem. The principle or least power works there (given a choice of solutions, use the least flexible/powerful to solve your problem).
I mean yes it's more verbose, BUT it's also super clear and obvious what things do, and they do not leak the underlying implementation.
The other key thing I think is not to over-engineer abstractions you don’t need yet. But to try and leave ‘seams’ where it’s obvious how to tease code about if you need to start building abstractions.
Can you say more about this?
I think I may have stumbled on a similar insight myself. In a side project (a roguelike game), I've been experimenting with a design that treats features as first-class, composable design units. Here is a list of the subfolder called game-features in the source tree:
actions
collision
control
death
destructibility
game-feature.lisp
hearing
kinematics
log
lore
package.lisp
rendering
sight
simulation
transform
"A Game Feature is responsible for providing components, events,
event handlers, queries and other utilities implementing a given
aspect of the game. It's primarily a organization tool for gameplay code.
Each individual Game Feature is represented by a class inheriting
from `SAAT/GF:GAME-FEATURE'. To make use of a Game Feature,
an object of such class should be created, preferably in a
system description (see `SAAT/DI').
This way, all rules of the game are determined by a collection of
Game Features loaded in a given game.
Game Features may depend on other Game Features; this is represented
through dependencies of their classes."
(Each feature also gets its set of integration tests that use synthetic scenarios to verify a particular aspect of the game works as I want it to.)
One negative side effect of this design is that the execution order of handlers for any given event is hard to determine from code. That's because, to have game features easily compose, GFs can request particular ordering themselves (e.g. "death" can demand its event handler to be executed after "destructibility" but before "log") - so at startup, I get an ordering preference graph that I reconcile and linearize (via topological sorting). I work around this and related issues by adding debug utilities - e.g. some extra code that can, after game startup, generate a PlantUML/GraphViz picture of all events, event handlers, and their ordering.
(I apologize for a long comment, it's a bit of work I always wanted to talk about with someone, but never got around to. The source of the game isn't public right now because I'm afraid of airing my hot garbage code.)
It might be hard to integrate related things, e.g. physical simulation/kinematics <- related to collisions, and maybe sight/hearing <- related to rendering; Which is all great if information flows one way, as a tree, but maybe complicated if it's a graph with intercommunication.
I thought about this before, and figured maybe the design could be initially very loose (and inefficient), but then a constraint-solver could wire things up as needed, i.e. pre-calculate concerns/dependencies.
Another idea, since you mention "logs" as a GF: AOP - using " join points" to declaratively annotate code. This better handles code that is less of a "module" (appropriate for functions and libraries) and more of a cross-cutting "aspect" like logging. This can also get hairy though: could you treated "(bad-path) exception handling" as an aspect? what about "security"?
Yes, it's all in Common Lisp.
> It might be hard to integrate related things, e.g. physical simulation/kinematics <- related to collisions, and maybe sight/hearing <- related to rendering;
It is, and I'm cheating a bit here. One simplification is that I'm writing a primarily text-based roguelike, so I don't have to bother with a lot of issues common to real-time 3D games. I can pick and choose the level of details I want to go (e.g. whether to handle collisions at a tile granularity, or to introduce sub-tile coordinates and maybe even some kind of object shape representation).
> Which is all great if information flows one way, as a tree, but maybe complicated if it's a graph with intercommunication.
The overall simulation architecture I'm exploring in this project is strongly event-based. The "game frame" is basically pulling events from a queue and executing them until the queue is empty, at which point the frame is considered concluded, and simulation time advances forward. It doesn't use a fixed time step - instead, when a simulation frame starts, the code looks through "actions" scheduled for game "actors" to find the one that will complete soonest, and moves the simulation clock to the completion time of that action. Then the action completion handler fires, which is the proper start of frame - completion handler will queue some events, and handlers of those events will queue those events, and the code just keeps processing events until the queue empties again, completing the simulation frame.
Structure-wise, simulation GF defines the concept of "start frame" and "end frame" (as events), "game clock" (as query) and ability to shift it (as event handler), but it's the actions GF that contains the computation of next action time. So, simulation GF knows how to tell and move time, but actions GF tells it where to move it to.
This is all supported by an overcomplicated event loop that lets GFs provide hints for handler ordering, but also separates each event handling process into four chains: pre-commit, commit, post-commit and abort. Pre-commit handlers fire first, filling event structure with data and performing validation. Then, commit handlers apply direct consequences of event to the real world - they alter the gameplay state. Then, post-commit handlers process further consequences of an event "actually happening". Alternatively, abort handlers process situations when an event was rejected during earlier chains. All of them can enqueue further events to be processed this frame.
So, for example, when you fire a gun, pre-commit handlers will ensure you're able to do it, and reject the event if you can't. If the event is rejected, abort chain will handle informing you that you failed to fire. Otherwise, the commit handlers will instantiate an appropriate projectile. Post-commit handlers may spawn events related to the weapon being fired, such as informing nearby enemies about the discharge.
This means that e.g. if I want to implement "ammunition" feature, I can make an appropriate GF that attaches a pre-commit handler to fire event - checking if you have bullets left and rejecting the event if you don't (events rejected in pre-commit stage are considered to "have never happened"), and a post-commit handler on the same event to decrease your ammo count. The GF is also responsible for defining appropriate components that store ammo count, so that (in classical ECS style) your "gun" entity can use it to keep track of ammunition. It also provides code for querying the current count, for other GFs that may care about it for some reason (and the UI rendering code).
> I thought about this before, and figured maybe the design could be initially very loose (and inefficient), but then a constraint-solver could wire things up as needed, i.e. pre-calculate concerns/dependencies.
I'm halfway there and I could easily do this, but for now opted against it, on the "premature optimization" grounds. That is, since all event handlers are registered when the actual game starts, I "resolve constraints" (read: convert sorting preferences into a DAG and toposort it; it's dumb and very much incorrect, but works well enough for now) and linearize handlers - so that during gameplay, each event handler chain (e.g. "fire weapon", pre-commit) is just a sequence of callbacks executed in order. It would be trivial to take such sequence, generate a function that executes them one by one (with the loop unrolled), and compile it down to metal - Common Lisp lets you do stuff like that - but I don't need it right now.
> Another idea, since you mention "logs" as a GF
FWIW, logs GF is implementing the user-facing logs typical for roguelike games - i.e. the bit that says "You open the big steel doors". Diagnostic logs I do classically.
> AOP - using " join points" to declaratively annotate code
In a way, my weird multi-chain event loop is a reimplementation of AOP. Method combinations in Common Lisp are conceptually similar too, but I'm not making big use of them in game feature-related code.
> This can also get hairy though: could you treated "(bad-path) exception handling" as an aspect? what about "security"?
Yeah, I'm not sure if this pattern would work for these - particularly in full-AOP, "inject anything anywhere" mode. I haven't tried it. Perhaps, with adequate tooling support, it's workable? Common Lisp is definitely not a language to try this in, though - it's too dynamic, so tooling would not be able to reliably tell you about arbitrary pointcuts.
In my case, I restricted the "feature-oriented design" to just game features - I feel it has a chance of working out, because in my mind, quite a lot of gameplay mechanics are conceptually additive. This project is my attempt at experimentally verifying if one could actually make a working game this way.
Your top issue in the runtime game loop is always with concurrency and synchronization logic - e.g. A spawns before B, if A's hitbox overlaps with B, is the first frame that a collision event occurs the frame of spawning or one frame after? That's the kind of issue that is hard to catch, occurs not often, and often has some kind of catastrophic impact if handled wrongly. But the actual effect of the event is usually a one-liner like "set a stun timer" - there is nothing to test with respect to the event itself! The perceived behavior is intimately coupled to when its processing occurs and when the effects are "felt" elsewhere in the loop - everything's tied to some kind of clock, whether it's the CPU clock, the rendered frame, turn-taking, or an abstracted timer. These kinds of bugs are a matter of bad specification, rather than bad implementation, so they resist automated testing mightily.
The most straightforward solution is, failing pure functions, to write more inline code(there is a John Carmack posting on inline code that I often use as a reference point). Enforce a static order of events as often as possible. Then debugging is always a matter of "does A happen before B?" It's there in the source code, and you don't need tooling to spot the issue.
The other part of this is, how do you load and initialize the scene? And that's a data problem that does call for more complex dependency management - but again, most games will aim to solve it statically in the build process of the game's assets, and reduce the amount of game state being serialized to save games, reducing the complexity surface of everything related to saves(versioning, corruption, etc). With a roguelike there is more of an impetus to build a lot of dynamic assets(dungeon maps, item placements etc.) which leads to a larger serialization footprint. But ultimately the focus of all of this is on getting the data to a place where you can bring it back up and run queries on it, and that's the kind of thing where you could theoretically use SQLite and have a very flexible runtime data model with a robust query system - but fully exploiting it wouldn't have the level of performance that's expected for a game.
Now, where can your system make sense? Where the game loop is actually dynamic in its function - i.e. modding APIs. But this tends to be a thing you approach gradually and grudgingly, because modders aren't any better at solving concurrency bugs and they are less incentivized to play nice with other mods, so they will always default to hacking in something that stomps the state, creating intermittent race conditions. So in practice you are likely to just have specific feature points where an API can exist(e.g. add a new "on hit" behavior that conditionally changes the one-liner), and those might impose some generalized concurrency logic.
The other thing that might help is to have a language that actually understands that you want to do this decoupling and has the tooling built in to do constraint logic programming and enforce the "musts" and "cannots" at source level. I don't know of a language that really addresses this well for the use case of game loops - it entails having a whole general-purpose language already and then also this other feature. Big project.
I've been taking the approach instead of aiming to develop "little languages" that compose well for certain kinds of features - e.g. instead of programming a finite state machine by hand for each type of NPC, devise a subcategory of state machines that I could describe as a one-liner, with chunks of fixed-function behavior and a bit of programmability. Instead of a universal graphics system, have various programmable painter systems that can manipulate cursors or selections to describe an image. The concurrency stays mostly static, but the little languages drive the dynamic behavior, and because they are small, they are easy to provide some tooling for.
I'll also add that this is an experimental hobby project, used to explore various programming techniques and architecture ideas, so I don't care about most constraints under which commercial game studios operate.
> The perceived behavior is intimately coupled to when its processing occurs and when the effects are "felt" elsewhere in the loop - everything's tied to some kind of clock, whether it's the CPU clock, the rendered frame, turn-taking, or an abstracted timer. These kinds of bugs are a matter of bad specification, rather than bad implementation, so they resist automated testing mightily.
Since day one of the project, the core feature was to be able to run headless automated gameplay tests. That is, input and output are isolated by design. Every "game feature" (GF) I develop comes with automated tests; each such test starts up a minimal game core with fake (or null) input and output, the GF under test, and all GFs on which it depends, and then executes faked scenarios. So far, at least for minor things, it works out OK. I expect I might hit a wall when there are enough interacting GFs that I won't be able to correctly map desired scenarios to actual event execution orders. We'll see what happens when I reach that point.
> that's the kind of thing where you could theoretically use SQLite and have a very flexible runtime data model with a robust query system - but fully exploiting it wouldn't have the level of performance that's expected for a game.
Funny you should mention that.
The other big weird thing about this project is that it uses SQLite for runtime game state. That is, entities are database rows, components are database tables, and the canonical gameplay state at any given point is stored in an in-memory SQLite database. This makes saving/loading a non-issue - I just use SQLite's Backup API to dump the game state to disk, and then read it back.
Performance-wise, I tested this approach extensively up front, by timing artificial reads and writes in expected patterns, including simulating a situation in which I pull map and entities data in a given range to render them on screen. SQLite turned out to be much faster than I expected. On my machine, I could easily get 60FPS out of that with minimum optimization work - but it did consume most of the frame time. Given that I'm writing a ASCII-style, turn(ish) roguelike, I don't actually need to query all that data 60 times per second, so this is quite acceptable performance - but I wouldn't try that with a real-time game.
> The other thing that might help is to have a language that actually understands that you want to do this decoupling and has the tooling built in to do constraint logic programming and enforce the "musts" and "cannots" at source level. I don't know of a language that really addresses this well for the use case of game loops - it entails having a whole general-purpose language already and then also this other feature. Big project.
Or a Lisp project. While I currently do constraint resolution at runtime, it's not hard to move it to compile time. I just didn't bother with it yet. Nice thing about Common Lisp is that the distinction between "compilation/loading" and "runtime" is somewhat arbitrary - any code I can execute in the latter, I can execute in the former. If I have a function that resolves constraints on some data structure and returns a sequence, and that data structure can be completely known at compile time, it's trivial to have the function execute during compilation instead.
> I've been taking the approach instead of aiming to develop "little languages" that compose well for certain kinds of features
I'm interested in learning more about the languages you developed - e.g. how your FSMs are encoded, and what that "programmable painter system" looks like. In my project, I do little languages too (in fact, the aforementioned "game features" are a DSL themselves) - Lisp makes it very easy to just create new DSLs on the fly, and to some extent they inherit the tooling used to power the "host" language.
When you create gameplay tests, one of the major limitations is in testing data. Many games end up with "playground" levels that validate the major game mechanics because they have no easier way of specifying what is, in essence, a data bug like "jump height is too short to cross gap". Now, of course you can engineer some kind of test, but it starts to become either a reiteration of the data (useless) or an AI programming problem that could be inverted into "give me the set of values that have solutions fitting these constraints" (which then isn't really a "test" but a redefinition of the medium, in the same way that a procedural level is a "solution" for a valid level).
It's this latter point that forms the basis of many of the "little languages". If you hardcode the constraints, then more of the data resides in a sweet spot by default and the runtime is dealing with less generality, so it also becomes easier to validate. One of my favorite examples of this is the light style language in Quake 1: https://quakewiki.org/wiki/lightstyle
It's just a short character string that sequences some brightness changes in a linear scale at a fixed rate. So it's "data," but it's not data encoded in something bulky like a bunch of floating point values. It's of precisely the granularity demanded by the problem, and much easier to edit as a result.
A short step up from that is something like MML: https://en.wikipedia.org/wiki/Music_Macro_Language - now there is a mostly-trivial parsing step involved, but again, it's "to the point" - it assumes features around scale and rhythm that allow it to be compact. You can actually do better than MML by encoding an assumption of "playing in key" and "key change" - then you can barf nearly any sequence of scale degrees into the keyboard and it'll be inoffensive, if not great music. Likewise, you could define rhythm in terms of rhythmic textures over time - sparse, held, arpeggiated, etc. - and so not really have to define the music note by note, making it easy to add new arrangements.
With AI, a similar thing can apply - define a tighter structure and the simpler thing falls out. A lot of game AI FSMs will follow a general pattern of "run this sequenced behavior unless something of a higher priority interrupts it". So encode the sequence, then hardcode the interruption modes, then figure out if they need to be parameterized into e.g. multiple sequences, if they need to retain a memory scratchpad and resume, etc. A lot of the headache of generalizing AI is in discovering needs for new scratchpads, if just to do something like a cooldown timer on a behavior or to retain a target destination. It means that your memory allocation per entity is dependent on how smart they have to be, which depends on the AI's program. It's not so bad if you are in something as dynamic as a Lisp, but problematic in the typical usages of ECS where part of the point is to systematize memory allocation.
With painting what you're looking for is a structuring metaphor for classes of images. Most systems of illustration have structuring metaphors of some kind specifically for defining proportions - they start with simple ratios and primitive shapes, and then use those as the construction lines for more detailed elements which subdivide the shapes again with another set of ratios. This is the conceptual basis of the common "6-8 heads of height" tip used in figure drawing - and there are systems of figure drawing which get really specific about what shapes to draw and how. If I encode such a system, I therefore have a method of automatic illustration that starts not with the actual "drawing" of anything, but with a proportion specification creating construction lines, which are then an input to a styling system that defines how to connect the lines or superimpose other shapes. Something I've been experimenting with to get those lines is a system that works by interpolation of coordinate transforms that aggregate a Cartesian and polar system together - e.g. I want to say "interpolate along this Cartesian grid, after it's been rotated 45 degrees". It can also perform interpolation between two entirely different coordinate systems(e.g. the same grid at two different scales). I haven't touched it in a while, but it generates interesting abstract animations, and I have a vision for turning that into a system for specifying character mannequins, textures, etc. Right now it's too complex to be a good one-liner system, but I could get there by building tighter abstractions on it in the same way as the music system.
My main thing this year has been a binary format that lets me break away from text encodings as the base medium, and instead have more precise, richer data types as the base cell type. This has gone through a lot of iteration to test various things I might want to encode and forms I could encode them in. The key thing I've hit on is to encode with a lot of "slack" in the system - each "cell" of data is 16 bytes; half of that is a header that contains information about how to render it, its ID in a listing of user named types, bitflags defined by the type, a "feature" value(an enumeration defined by the type), and a version field which could be used for various editing features. The other half is a value, which could be a 64-bit value, 8 bytes, a string fragment, etc. - the rendering information field indicates what it is in those general terms, but the definite meaning is named by the user type. The goal is to use this as a groundwork to define the little languages further - rather than relying on "just text" and sophisticated parsing, the parse is trivialized by being able to define richer symbols - and then I can provide more sophisticated editing and visualization more easily. Of course, I'm placing a bet on either having an general-purpose editor for it that's worthwhile, or being able to define custom editors that trivialize editing, neither of which might pan out; there's a case for either "just text" or "just spreadsheets" still beating my system. But I'd like to try it, since I think this way of structuring the bits is likely to be more long-run sustainable.
I don't think I can respond properly and ask the many more questions I have in a HN thread that's already this aged, so if you'd like to talk more, feel free to hit me up (my contact details are in my profile).
My clean code book:
* Put logic closest to where it needs to live (feature folders)
* WET (Write everything twice), figure out the abstraction after you need something a 3rd time
* Realize there's no such thing as "clean code"