That and referential transparency are huge wins. It is a shame languages like Python make it such a challenge.
For example, let's assign a constant variable to a sequence; should you do use list or tuple (FOO = [1,2,3] or FOO = (1,2,3) )? It doesn't matter, the constant capitalization discourages you to mutate it.
It's strange that a general idea expressing (assuming?) overarching maturity has led to quite so many petulant arguments, wouldn't you say @ketozhang?
So now I'm more confused than when I initially read your comment. Do you mean "referential transparency" as in "expressions without side effects" or do you mean it in some other way that I don't understand yet?
Python code (pre-v3) that branches cannot be referentially transparent by definition because runtime context (the state of True/False bindings) is a hidden input in every boolean expression. You could have millions of lines of side effect free code and it will break completely if that one statement is run before the rest of the code.
Programmers depend on the referential transparency of keywords like "true", "false", "for", etc whether they're writing pure functional code or imperative spaghetti messes.
(Specifically avoid namedtuples unless you're hoisting a tuple to a real record. Record types also being a weird union with a tuple is...oft confusing)
For immutable basic data types, adopt type annotations and prefer to use sequence/iterable and mapping to direct didn't/list types. These provide statically verifiable immutability and also allow begin to encourage useful async-friendly patterns.
[1]: https://www.python.org/dev/peps/pep-0591/For non-primitive members, `frozenset` is in the standard library, but frozen dictionaries are unfortunately lacking.
Would Thursday be a good time to checkpoint on this?
Thank you so much.
but what's the point then? why spend effort for it, if it isn't expected to be better?
Learning. The result may not be meaningfully better, but you gain fuller understanding of the problem domain. The third rewrite might have a chance of being an improvement ;).
Somewhat relatedly, I've refactored processes down from 25 hours to 20 minutes. It got to 20 minutes, and other people started nitpicking that it could 'be faster' (the same people who'd let it get to 25 hours, then threw up their hands and said "it can't be fixed"). They spent loads of time getting it from 20 minutes down to... 17-18 minutes (spending 2+ weeks in the process). I'd indicated "there's really no more 'there' there - this is about as fast as it's going to get, without faster hardware". They had to find out for themselves there wasn't much more juice left in the tank.
Maybe it's not related, but I felt like sharing it anyway... :)
20 to 17 minutes is a 15% time savings still.
In all seriousness, from a business standpoint, it's almost never worth rewriting anything unless you're already making significant changes to that code. The opportunity cost of not spending that time building things that get you paid is just too high—not to mention all the new bugs that you're likely to introduce.
In my experience, most of the insights cluster in two places: just as you start writing your solution, and just as you think you're done with it. So you want to get through the whole process quickly, just to learn the things you need to write a proper solution.
Related, from a different HN thread - https://news.ycombinator.com/item?id=27692710 - "action produces information".
that's not really what a rewrite is. A rewrite is when something has already been in production, and the engineer re-do the same feature set, but refactored or rewritten in a different (possibly better, but not guaranteed) form. This rewrite takes up the time that would've been spent on something else.
If you were doing a "rewrite" during implementation, but still delivered on time, that's not a rewrite - that's just good engineering, doing exploratory coding! If you couldn't deliver on time because of the rewrite, then i guess you can call it a rewrite.
Throwing away a whole repository that was tested and running in production or on some dev environment, is something different in my point of view. And I think that's what people mean by "rewrite".
You are also assuming one just rewrites stuff blindly and randomly without any prior new insight.
> 4. Delete code, delete tests, re-write stuff, re-write it again. Painful? Keep doing this till you get to the other side and feel liberation and empowerment, took me years to get past wincing at the idea of re-writing that thing AGAIN
With experience I've realized that rewrites are almost never a good idea[0]. They are time consuming and inevitably introduce new bugs (or reintroduce old ones), and the benefit is almost always marginal. If your architecture allows it, it's better to sandbox working legacy code and leave it as-is than constantly rewrite it.
My own policy is: if you're making significant changes to a unit of code anyway, clean it up or rewrite it; otherwise, make the smallest change necessary and leave it alone. If it ain't broke, don't fix it.
(Obligatory Joel Spolsky article on the topic that everyone has probably already read: https://www.joelonsoftware.com/2000/04/06/things-you-should-...)
[0]: ...with a possible exception if you're at a very large tech company with a lot of resources, where the testing/QA processes are a lot more thorough and it's possible to keep up with constant changes like this. But even then, seeing the amount of bugs introduced with each update to Facebook or Gsuite or any other large piece of tech, I'm skeptical.
What if your architecture is the problem? If you got the domain model wrong and you have deeply-nested complex types used throughout, you will never break free without a clean sheet rewrite. Certainly, you can refactor bits and pieces to conform to a correct model, but it's going to be an uphill battle the whole way. 10x if you are already in production with business state tied to the legacy schema(s).
Most of the horrible things I have seen at code review time have a root cause somewhere in poor domain modeling.
For example: Someone put support for 2 customer addresses as facts directly in the Customer type, so now you can't deal with the new edge case of 5+ addresses per customer, or model the idea that the address might be shared between more than 1 customer (and/or some other business types).
If you didn't model for 3NF/BCNF/DKNF up front, you might as well start over from the beginning in my experience. If your problem domain is not that complex, you can probably survive with something really shitty, but the moment you enter into 50+ types, 1000+ facts and 100+ relations, things are impossible to manage without strong discipline in this area.
I'd still prefer to do it incrementally if possible.
> if you're making significant changes to a unit of code anyway, clean it up or rewrite it; otherwise, make the smallest change necessary and leave it alone. If it ain't broke, don't fix it.
Don't be afraid to rewrite something when it needs it, and build knowing it's very possible you'll rewrite later.
If there are good unit tests, and it's a sufficiently small/decoupled piece of code, then rewriting is not so bad. These are all self-reinforcing things: small, decoupled code tends to be easy to rewrite; Testable code tends to be decoupled.
Systems always get more complex over time. The key thing is figuring out when your simple component is starting on the path to getting too complex, and taking the time to rewrite it as early as possible -- maybe it should be two smaller components, or maybe the entire organization of that are needs to be different. If you wait, and just keep adding "small" things, eventually you have a monster that's an order of magnitude more complex to deal with.
The other bit of this is writing code knowing you can (and may likely) rewrite it later. If you try to predict the future complexity early in design, 9 times out of 10 you will get it wrong, and you'll end up in a lose-lose situation: you have a overly-complex component to deal with, and it ends up needing a rewrite later anyway. Rewriting this is even harder because you have to undo the unnecessary complexity. This is also known as YAGNI ("You Aren't Gonna Need It").
The problem with this mentality is that everyone thinks they write “good unit tests”, but bugs are still found in production. :) You can’t use unit tests as justification for software being reliable; being battle tested in the real world is a much better indicator in practice.
I mostly agree with your other points, although I would still advocate doing these rewrites in as small of pieces as possible, in a way that’s as backwards compatible as possible, instead of all at once. I think you may be saying roughly the same thing, but this thread has shown that people have different definitions of “rewrite”, so it’s hard to tell. :)
My point on unit tests to support a rewrite is that it can catch a lot of edge cases, and you can remain reasonably certain you didn't break things if tests are still passing when you're done.
But otherwise I totally agree -- you can have as many tests as you want, with whatever coverage numbers you want, and it will still fall down in the real world. I don't focus on coverage or being dogmatic about TDD. I like to heavily unit test algorithms/regexes/etc -- anything with a defined input and output. At the same time, I hate testing 'glue' code (like an MVC controller), and would way prefer to rewrite it to be so simple it either works or it doesn't work at all (causing a big, obvious failure).
As bugs are found in the real world, in the ideal case write a test which basically guarantees that bug never happens again. Not always possible or takes such a massive effort that it isn't worth it, but usually it pays off.
> I would still advocate doing these rewrites in as small of pieces as possible, in a way that’s as backwards compatible as possible, instead of all at once.
Yes, exactly, that's what I was trying to get at. Big rewrites are hard to get approval/agreement to start, hard to actually do and often unsuccessful. Worse, a team with the mindset "ah, it's okay if we cut corners here, we're going to rewrite this whole thing someday" will rack up a serious amount of technical debt -- and make that big rewrite even harder. Ironically this makes a big rewrite even harder to start and succeed, but also more necessary.
Small continuous rewrites are a way of constantly improving quality, while avoiding a lot of these traps.
In some ways software engineering is so immature compared with other engineering disciplines.
I believe more in boy scout - "if you touch it leave it better than when you arrived".
:-)
It's not really about better or worse, what I've found is different people have different backgrounds and what principles they find useful is based on several things, which may be immuntable - such as strongly held values.
You're never going to make me not care about aesthetics for example, as that is intrinsic to me and that will affect the principles I like and ultimately the people that I work best with.
An analogy I like to think about is that people with very different principles are like two people holding a rope and pulling away from each other - you aren't going to get anywhere fast.
Whereas people with similish principles, will generally go in the right direction. Sure, they'll get tangled up from time to time and you won't always want to get in exactly the same way. But there's a collaborative nature to it and you'll both improve.
That sounds wrong. As in "I could upload quotes from Gang Of Four books and claim I am the source" wrong.
You can't be an author if you aren't the author of a principle or the principle is too generic. If the principle is already open source (e.g. on wikipedia, has a creative commons license) you can submit it but not claim you are the author for it and submit it under the same licensing terms (CC-BY-SA) as long as it doesn't break the license.
Codifying the principle for the first time takes effort and people can iterate on it to make it better over time. Many people may have had similar thoughts before, but if it's not a general principle already being used the first to turn it into a principle - to put a stake in the ground - benefits everyone and can help improve everyone's capability.
I believe the author should be rewarded for that effort, as long as it is their own unique work.
Then being able to create your own lists for unique situations. Say "Lupire's CTO list" or "Lupire's management principles" and to share that with your team or as a reminder to yourself.
Of course you should always be able to export it and put it in a format that's useful to you. And that's been really important to the design. From using markdown format to embedding license information and meta data with the principle, it should help a lot with portability.
What I've come to realize is that whenever people talk about wanting immutability, they really want value semantics. Value semantics can be achieved without immutability - the best example is Swift's structs. They can be both mutable or immutable, but they can never be shared, i.e. there can never be more than one reference to a single struct value. This means that modifying the value has no effect on other references, because there can be no other references.
This makes whether or not the value is immutable irrelevant, or at least not necessary. You get the same benefits - you can reason about your code locally and not worry about your changes modifying unknown parts of the program. Local reasoning is the end goal, and value semantics is the mechanism of getting to that goal. Immutability is only one way of getting value semantics.
Is that really worth it?
Also, creating the second vector should really look like "v2 = v.copy(x = x + 5); return v2.length". Or even just "return v.copy(x = x + 5).length".
This is the same model that Rust takes too. It doesn't eliminate mutability, only controls it. The argument that "performance doesn't matter anymore because computers are so fast" is a bad one. Efficiency is efficiency, and immutability will always be less efficient.
func newStructValue(s: Struct) -> Struct {
s.value = 5
return s
}
Also, don't let people guess what are those commands - create a menu that is run as soon as you clone the repo and shows all commands. I now use Invoke-Build [1] system for that and on all projects you type ib ? to get that list and later ib <task> to execute it (or any combination of it, i.e. ib DropDatabase, Run, Test). Basic tasks are named the same on all projects no matter the underlying technology.
Later you use the same command on CI server - its just another developer that runs ib Build, Test, Publish. Any more programming in CI yaml files then 1 to 3 lines per job is so wrong.
Can you give some examples?
Similarly, many systems involve consuming from some message queue. It's basically impossible to guarantee exactly-once delivery in most systems. You either have to risk missing a message, or having it delivered multiple times. If you're running idempotent code, you can always err on the side of redundant delivery without any ill effects.
My immediate thought was that some order processing step somewhere is not idempotent.
You should still try and make most things idempotent.
Let's say you're controlling a factory and you have a function that fills a reservoir. A naive way would be to define the semantics of the operation as "send enough liquid" and open the circuit for 10 minutes when activated. Since it takes 10 min to fill, all is good.
But what if your program crashes and you have to rerun. If the reservoir was already semi-filled, you will overflow it.
If the semantics of the operation is "send liquid until 'full' sensor indicates you're done", that's liberating. You no longer have to worry about overflowing the reservoir.
Reading a variable shared between threads without a lock is side-effect free but not idempotent.
If we’re talking engineering principles, not team dynamics, then it’s:
If we’re talking softer stuff: